Polling print devices using SNMP while the printer is printing



I’ll tell you my experience in creating an application that collects statistics about printing devices using the SNMP protocol. In total, more than 1000 devices and the survey is performed in real time while the printer is printing. How this is all implemented, you will learn from the article.



Introduction


SNMP is a standard Internet protocol for managing devices in IP networks based on TCP / UDP architectures.

Using SNMP does not collect information about the user who sent the file to print. In addition, SNMP cannot be used for local printers connected via USB.
To solve these problems, you need to create a client that must collect information and send all statistics to the server. The organization was so established that the same Linux OS was installed on all client machines. Accordingly, the client for collecting print statistics is written in the Linux system as a daemon (an analogue of the service in Windows).

Print Information Collection Client


The print collection client (written in Python) performs the following tasks:

  1. intercepts messages from the CUPS print service
  2. saves print information to a CSV file
  3. sends CSV files to a folder on the server via FTP

The CSV file contains the following fields:

  1. Date and time of print job
  2. IP address or domain name of the computer from which the document was sent to print
  3. IP address or host of the print device
  4. Cups Job ID - Job ID from the CUPS Print Service
  5. Job name - this is most often the file name
  6. Name of user who sent to print
  7. Number of pages
  8. Print job size (Job size in KB)
  9. Print Status (Job Status)
  10. Number of copies (Number of copies)
  11. The number of copies on the job file - is used to more accurately determine the number of copies;
  12. Number of pages by SNMP protocol - used to more accurately determine the number of pages
  13. SNMP is available - this is a sign that the device should be continuously polled using the SNMP protocol while the document is being printed
  14. The counter of the printing device at the time of printing start (SNMP variable value) is the verification counter that determines the start of SNMP polling by the server

The client does not have a GUI interface, all settings are registered in the configuration file

Configuration File Example
[main]
#    
app_directory = /opt/printwatcher

#  .    
#        )
stat_file_directory = /opt/printwatcher/data

# ,      
state_file = /opt/printwatcher/data/state.pickle

# ,       
command_state_file = /opt/printwatcher/data/command.state

#    
#   :
#  datetime -     ISO 8601
#  random -    8 
stat_filename_template = {random}_print_jobs.csv

#      
# (    )
max_stat_file_size_mb = 10

#  CSV.  : TAB, SPACE, SEMICOLON, COMMA
csv_delimiter = SEMICOLON

#      
start_service = start


[logging]

#   (DEBUG, INFO, TRACE)
level = DEBUG

#  
log_file = /var/log/printwatcher.log

#      
restart_log_file = /var/log/printwatcher-restart.log

#   
# (  /   logrotate)
max_log_file_size_mb = 10


[cups]

# ,      CUPS
spool_directory = /var/spool/cups/

#    CUPS    
# ( )
job_file_lifetime_hours = 48

#           
# ( )
job_info_lifetime_hours = 48


#  FTP
[ftp]

host = 192.168.1.39
user = ftpuser
password = P@ssw0rd
port = 21

#         FTP
# ( )
connection_retry_seconds = 10

#  ,       FTP
# ( )
connection_timeout_seconds = 3600

#        FTP 
# ( )
polling_interval = 30

#     ftp-
management_directory = /management

#       ftp-
response_directory = /management

#   
update_directory = /update

#      
data_directory = /data

#  SNMP
[snmp]
#      SNMP
retrieve_snmp_info = false
community = public
oid = 1.3.6.1.2.1.43.10.2.1.4.1.1
port = 161


So, the client is written, now we need a server that will not only collect information from all clients, but also the server must monitor emergency situations - pause printing, missing or jammed paper, etc.

Print Information Collection Server Architecture


The server is written in C #, the DBMS is MSSQL.

  1. Database Configurator is a GUI application that allows you to create and delete a database.
    The database is maximally normalized so that there are no text fields in the main statistics table to save disk space.
  2. , GUI SNMP-, OID, .



  3. ( Windows)



  1. CSV-
  2. CSV
  3. CSV-
  4. CSV- ( ) [ ]
  5. CSV-
  6. OID'
  7. ,


Before the service will monitor printing in real time, it is necessary to parse the old CSV files (which accumulated until the server was working) and save the information in the database. We will call this point “the first reading of the CSV file directory”, which can help quickly collect statistics of the old print and save it to the database.

Then the service switches to constant monitoring of the appearance of CSV files in a folder in a separate stream with an interval of 1 second. Each CSV file is processed in a separate stream.

If there are a lot of CSV files in the folder, then in this case there is a limit on the number of simultaneous read streams of files so that the application works stably. When the service starts, there can be a lot of CSV files in the folder and these files are processed without SNMP polling; accordingly, the first reading of the directory has a separate limit on the number of CSV file processing threads (it was experimentally established that the number of threads on the first reading of CSV files should be much smaller than the number of threads after processing a large number of files).

The algorithm provides for determining the IP address by the device name and vice versa (determining the device name by IP address) using the System.Net.Dns.GetHostEntry () method. If there is no connection with the device, the System.Net.Dns.GetHostEntry () method takes a very long time, in connection with this, the client that provides this information to CSV has been finalized. But this check remained on the server and the System.Net.Dns.GetHostEntry () method is still a bottleneck.

Saving information in the database


In addition to information about printing, the database stores information for statistics:

  1. SNMP polling start and end time
  2. total number of lines in the CSV file
  3. The number of lines with SNMP polling enabled.

A list of devices that should not be polled is provided for, so-called exceptions for which nothing is stored in the database.

After reading the entire CSV file, information about all print jobs is collected in the List object, which should be monitored by SNMP polling. After reading the CSV file, it is transferred to the archive, if specified by the settings, and then the CSV file is deleted.

After the CSV file is fully read, it is checked that the polling settings are configured using the SNMP protocol, then the devices are sorted by priority:

  1. The default group, in which there are all devices of the ricoh_dmnx base, is the name of the database of the monitoring system from the manufacturer Ricoh, from which we take the list of devices to be polled.
  2. Devices that are in the ricoh_dmnx database, but not in groups
  3. Devices that are in the ricoh_dmnx database and are in groups
  4. Devices that are not in the ricoh_dmnx database, but are in our service database

Each device has a print queue so that at one point in time, an SNMP poll works with only one print job. Thus, while the device is being interrogated using the SNMP protocol, the user can send another file for printing and a new CSV file can be received on the server, which is saved immediately in the database, but the poll will be performed after the current document.

SNMP polling tracks changes in the print counter, device status, and a sign of a device error during printing. The latency between SNMP polls is 300 ms.

Underwater rocks


The server algorithm has been constantly improved, as there were many different situations in which the counter was considered incorrectly. Here are some tips from personal experience:

1) Do not continue polling while the device is in an error state (for example: paper is jammed)

2) A printer error may change over time, so the SNMP polling timeout should not work if the error changes (for example, the printer ran out of paper, but after adding paper to the tray, the printer status changed)

3) If several jobs are sent to the print device at the same time, then the counter after printing the first job:
- does not change the status
- changes the counter

System OIDs of Printing Devices


Based on the analysis of these three OIDs, my application for polling SNMP devices was written:

1) [1.3.6.1.2.1.43.10.2.1.4.1.1] - Total device counter

2) [1.3.6.1.2.1.25.3.5.1 .1.1] - Device status



3) [1.3.6.1.2.1.25.3.2.1.5.1] - Device error during printing

Reports


Reports are implemented using SQL Server Reporting Services.

Monthly Print Comparison Report




Printed Jobs by User




SNMP Request Summary Report




SNMP Request Statistics




Conclusion


I was glad to share my experience in implementing one of my most complex applications, which has been successfully operating since 2018.

A peculiarity in the implementation of the print collection project was that I did not have a single printer with me. Worked through an emulator that can be quickly installed and configured (it is enough to have a MIB file for the settings of the printing device).

In the future, I will write about how the management of print agents was done.

Source: https://habr.com/ru/post/undefined/


All Articles