1 INN Python Filtering and Authentication Support
3 This file documents INN's built-in optional support for Python article
4 filtering. It is patterned after the Perl and (now obsolete) TCL hooks
5 previously added by Bob Heiney and Christophe Wolfhugel.
7 For this filter to work successfully, you will need to have at least
8 Python 1.5.2 installed. You can obtain it from
9 <http://www.python.org/>.
11 The innd Python interface and the original Python filtering
12 documentation were written by Greg Andruk (nee Fluffy)
13 <gerglery@usa.net>. The Python authentication and authorization support
14 for nnrpd as well as the original documentation for it were written by
15 Ilya Etingof <ilya@glas.net> in December 1999.
19 Once you have built and installed Python, you can cause INN to use it by
20 adding the --with-python switch to your "configure" command. You will
21 need to have all the headers and libraries required for embedding Python
22 into INN; they can be found in Python development packages, which
23 include header files and static libraries.
25 You will then be able to use Python authentication, dynamic access group
26 generation and dynamic access control support in nnrpd along with
27 filtering support in innd.
29 See the ctlinnd(8) manual page to learn how to enable, disable and
30 reload Python filters on a running server (especially "ctlinnd mode",
31 "ctlinnd python y|n" and "ctlinnd reload filter.python 'reason'").
33 Also, see the filter_innd.py, nnrpd_auth.py, nnrpd_access.py and
34 nnrpd_dynamic.py samples in your filters directory for a demonstration
35 of how to get all this working.
37 Writing an innd Filter
41 You need to create a filter_innd.py module in INN's filter directory
42 (see the *pathfilter* setting in inn.conf). A heavily-commented sample
43 is provided; you can use it as a template for your own filter. There is
44 also an INN.py module there which is not actually used by INN; it is
45 there so you can test your module interactively.
47 First, define a class containing the methods you want to provide to
48 innd. Methods innd will use if present are:
51 Not explicitly called by innd, but will run whenever the filter
52 module is (re)loaded. This is a good place to initialize constants
53 or pick up where "filter_before_reload" or "filter_close" left off.
55 filter_before_reload(*self*)
56 This will execute any time a "ctlinnd reload all 'reason'" or
57 "ctlinnd reload filter.python 'reason'" command is issued. You can
58 use it to save statistics or reports for use after reloading.
61 This will run when a "ctlinnd shutdown 'reason'" command is
64 filter_art(*self*, *art*)
65 *art* is a dictionary containing an article's headers and body.
66 This method is called every time innd receives an article. The
67 following can be defined:
69 Also-Control, Approved, Bytes, Cancel-Key, Cancel-Lock,
70 Content-Base, Content-Disposition, Content-Transfer-Encoding,
71 Content-Type, Control, Date, Date-Received, Distribution, Expires,
72 Face, Followup-To, From, In-Reply-To, Injection-Date, Injection-Info,
73 Keywords, Lines, List-ID, Message-ID, MIME-Version, Newsgroups,
74 NNTP-Posting-Date, NNTP-Posting-Host, Organization, Originator,
75 Path, Posted, Posting-Version, Received, References, Relay-Version,
76 Reply-To, Sender, Subject, Supersedes, User-Agent,
77 X-Auth, X-Canceled-By, X-Cancelled-By, X-Complaints-To, X-Face,
78 X-HTTP-UserAgent, X-HTTP-Via, X-Mailer, X-Modbot, X-Modtrace,
79 X-Newsposter, X-Newsreader, X-No-Archive, X-Original-Message-ID,
80 X-Original-Trace, X-Originating-IP, X-PGP-Key, X-PGP-Sig,
81 X-Poster-Trace, X-Postfilter, X-Proxy-User, X-Submissions-To,
82 X-Trace, X-Usenet-Provider, Xref, __BODY__, __LINES__.
84 Note that all the above values are as they arrived, not modified by
85 your INN (especially, the Xref: header, if present, is the one of
86 the remote site which sent you the article, and not yours).
88 These values will be buffer objects holding the contents of the same
89 named article headers, except for the special "__BODY__" and
90 "__LINES__" items. Items not present in the article will contain
93 "art('__BODY__')" is a buffer object containing the article's entire
94 body, and "art('__LINES__')" is an int holding innd's reckoning of
95 the number of lines in the article. All the other elements will be
96 buffers with the contents of the same-named article headers.
98 The Newsgroups: header of the article is accessible inside the
99 Python filter as "art['Newsgroups']".
101 If you want to accept an article, return "None" or an empty string.
102 To reject, return a non-empty string. The rejection strings will be
103 shown to local clients and your peers, so keep that in mind when
104 phrasing your rejection responses.
106 filter_messageid(*self*, *msgid*)
107 *msgid* is a buffer object containing the ID of an article being
108 offered by IHAVE or CHECK. Like with "filter_art", the message will
109 be refused if you return a non-empty string. If you use this
110 feature, keep it light because it is called at a rather busy place
111 in innd's main loop. Also, do not rely on this function alone to
112 reject by ID; you should repeat the tests in "filter_art" to catch
113 articles sent with TAKETHIS but no CHECK.
115 filter_mode(*self*, *oldmode*, *newmode*, *reason*)
116 When the operator issues a ctlinnd "pause", "throttle", "go",
117 "shutdown" or "xexec" command, this function can be used to do
118 something sensible in accordance with the state change. Stamp a log
119 file, save your state on throttle, etc. *oldmode* and *newmode*
120 will be strings containing one of the values in ("running",
121 "throttled", "paused", "shutdown", "unknown"). *oldmode* is the
122 state innd was in before ctlinnd was run, *newmode* is the state
123 innd will be in after the command finishes. *reason* is the comment
124 string provided on the ctlinnd command line.
126 How to Use these Methods with innd
128 To register your methods with innd, you need to create an instance of
129 your class, import the built-in INN module, and pass the instance to
130 "INN.set_filter_hook". For example:
133 def filter_art(self, art):
138 def filter_messageid(self, id):
145 INN.set_filter_hook(myfilter)
147 When writing and testing your Python filter, don't be afraid to make use
148 of "try:"/"except:" and the provided "INN.syslog" function. stdout and
149 stderr will be disabled, so your filter will die silently otherwise.
151 Also, remember to try importing your module interactively before loading
152 it, to ensure there are no obvious errors. One typo can ruin your whole
153 filter. A dummy INN.py module is provided to facilitate testing outside
154 the server. To test, change into your filter directory and use a
157 python -ic 'import INN, filter_innd'
159 You can define as many or few of the methods listed above as you want in
160 your filter class (it is fine to define more methods for your own use;
161 innd will not be using them but your filter can). If you *do* define
162 the above methods, GET THE PARAMETER COUNTS RIGHT. There are checks in
163 innd to see whether the methods exist and are callable, but if you
164 define one and get the parameter counts wrong, innd WILL DIE. You have
165 been warned. Be careful with your return values, too. The "filter_art"
166 and "filter_messageid" methods have to return strings, or "None". If
167 you return something like an int, innd will *not* be happy.
169 A Note regarding Buffer Objects
171 Buffer objects are cousins of strings, new in Python 1.5.2. Using
172 buffer objects may take some getting used to, but we can create buffers
173 much faster and with less memory than strings.
175 For most of the operations you will perform in filters (like
176 "re.search", "string.find", "md5.digest") you can treat buffers just
177 like strings, but there are a few important differences you should know
180 # Make a string and two buffers.
185 s == bs # - This is false because the types differ...
186 buffer(s) == bs # - ...but this is true, the types now agree.
187 s == str(bs) # - This is also true, but buffer() is faster.
188 s[:2] == bs[:2] # - True. Buffer slices are strings.
190 # While most string methods will take either a buffer or a string,
191 # string.join (in the string module) insists on using only strings.
193 string.join([str(b), s], '.') # Returns 'def.abc'.
194 '.'.join([str(b), s]) # Returns 'def.abc' too.
195 '.'.join([b, s]) # This raises a TypeError.
197 e = s + b # This raises a TypeError, but...
199 # ...these two both return the string 'abcdef'. The first one
200 # is faster -- choose buffer() over str() whenever you can.
204 g = b + '>' # This is legal, returns the string 'def>'.
206 Functions Supplied by the Built-in innd Module
208 Besides "INN.set_filter_hook" which is used to register your methods
209 with innd as it has already been explained above, the following
210 functions are available from Python scripts:
212 addhist(*message-id*)
213 article(*message-id*)
215 havehist(*message-id*)
218 newsgroup(*groupname*)
219 syslog(*level*, *message*)
221 Therefore, not only can innd use Python, but your filter can use some of
222 innd's features too. Here is some sample Python code to show what you
223 get with the previously listed functions.
227 # Python's native syslog module isn't compiled in by default,
228 # so the INN module provides a replacement. The first parameter
229 # tells the Unix syslogger what severity to use; you can
230 # abbreviate down to one letter and it's case insensitive.
231 # Available levels are (in increasing levels of seriousness)
232 # Debug, Info, Notice, Warning, Err, Crit, and Alert. (If you
233 # provide any other string, it will be defaulted to Notice.) The
234 # second parameter is the message text. The syslog entries will
235 # go to the same log files innd itself uses, with a 'python:'
237 syslog('warning', 'I will not buy this record. It is scratched.')
239 vehicle = 'hovercraft'
240 syslog('N', 'My %s is full of %s.' % (vehicle, animals))
242 # Let's cancel an article! This only deletes the message on the
243 # local server; it doesn't send out a control message or anything
244 # scary like that. Returns 1 if successful, else 0.
245 if INN.cancel('<meow$123.456@solvangpastries.edu>'):
250 # Check if a given message is in history. This doesn't
251 # necessarily mean the article is on your spool; cancelled and
252 # expired articles hang around in history for a while, and
253 # rejected articles will be in there if you have enabled
254 # remembertrash in inn.conf. Returns 1 if found, else 0.
255 if INN.havehist('<z456$789.abc@isc.org>'):
256 comment = "*yawn* I've already seen this article."
258 comment = 'Mmm, fresh news.'
260 # Here we are running a local spam filter, so why eat all those
261 # cancels? We can add fake entries to history so they'll get
262 # refused. Returns 1 on success, 0 on failure.
263 cancelled_id = buffer('<meow$123.456@isc.org>')
264 if INN.addhist("<cancel." + cancelled_id[1:]):
265 thought = "Eat my dust, roadkill!"
267 thought = "Darn, someone beat me to it."
269 # We can look at the header or all of an article already on spool,
270 # too. Might be useful for long-memory despamming or
271 # authentication things. Each is returned (if present) as a
272 # string object; otherwise you'll end up with an empty string.
273 artbody = INN.article('<foo$bar.baz@bungmunch.edu>')
274 artheader = INN.head('<foo$bar.baz@bungmunch.edu>')
276 # As we can compute a hash digest for a string, we can obtain one
277 # for artbody. It might be of help to detect spam.
278 digest = INN.hashstring(artbody)
280 # Finally, do you want to see if a given newsgroup is moderated or
281 # whatever? INN.newsgroup returns the last field of a group's
282 # entry in active as a string.
283 froupflag = INN.newsgroup('alt.fan.karl-malden.nose')
285 moderated = 'no such newsgroup'
286 elif froupflag == 'y':
288 elif froupflag == 'm':
291 moderated = "something else"
293 Writing an nnrpd Filter
295 Changes to Python Authentication and Access Control Support for nnrpd
297 The old authentication and access control functionality has been
298 combined with the new readers.conf mechanism by Erik Klavon
299 <erik@eriq.org>; bug reports should however go to <inn-bugs@isc.org>,
302 The remainder of this section is an introduction to the new mechanism
303 (which uses the *python_auth*, *python_access*, and *python_dynamic*
304 readers.conf parameters) with porting/migration suggestions for people
305 familiar with the old mechanism (identifiable by the now deprecated
306 *nnrpperlauth* parameter in inn.conf).
308 Other people should skip this section.
310 The *python_auth* parameter allows the use of Python to authenticate a
311 user. Authentication scripts (like those from the old mechanism) are
312 listed in readers.conf using *python_auth* in the same manner other
313 authenticators are using *auth*:
315 python_auth: "nnrpd_auth"
317 It uses the script named nnrpd_auth.py (note that ".py" is not present
318 in the *python_auth* value).
320 Scripts should be placed as before in the filter directory (see the
321 *pathfilter* setting in inn.conf). The new hook method "authen_init"
322 takes no arguments and its return value is ignored; its purpose is to
323 provide a means for authentication specific initialization. The hook
324 method "authen_close" is the more specific analogue to the old "close"
325 method. These two method hooks are not required, contrary to
326 "authenticate", the main method.
328 The argument dictionary passed to "authenticate" remains the same,
329 except for the removal of the *type* entry which is no longer needed in
330 this modification and the addition of several new entries (*port*,
331 *intipaddr*, *intport*) described below. The return tuple now only
332 contains either two or three elements, the first of which is the NNTP
333 response code. The second is an error string which is passed to the
334 client if the response code indicates that the authentication attempt
335 has failed. This allows a specific error message to be generated by the
336 Python script in place of the generic message "Authentication failed".
337 An optional third return element, if present, will be used to match the
338 connection with the *user* parameter in access groups and will also be
339 the username logged. If this element is absent, the username supplied
340 by the client during authentication will be used, as was the previous
343 The *python_access* parameter (described below) is new; it allows the
344 dynamic generation of an access group of an incoming connection using a
345 Python script. If a connection matches an auth group which has a
346 *python_access* parameter, all access groups in readers.conf are
347 ignored; instead the procedure described below is used to generate an
348 access group. This concept is due to Jeffrey M. Vinocur and you can add
349 this line to readers.conf in order to use the nnrpd_access.py Python
350 script in *pathfilter*:
352 python_access: "nnrpd_access"
354 In the old implementation, the authorization method allowed for access
355 control on a per-group basis. That functionality is preserved in the
356 new implementation by the inclusion of the *python_dynamic* parameter in
357 readers.conf. The only change is the corresponding method name of
358 "dynamic" as opposed to "authorize". Additionally, the associated
359 optional housekeeping methods "dynamic_init" and "dynamic_close" may be
360 implemented if needed. In order to use nnrpd_dynamic.py in
361 *pathfilter*, you can add this line to readers.conf:
363 python_dynamic: "nnrpd_dynamic"
365 This new implementation should provide all of the previous capabilities
366 of the Python hooks, in combination with the flexibility of readers.conf
367 and the use of other authentication and resolving programs (including
368 the Perl hooks!). To use Python code that predates the new mechanism,
369 you would need to modify the code slightly (see below for the new
370 specification) and supply a simple readers.conf file. If you do not
371 want to modify your code, the sample directory has
372 nnrpd_auth_wrapper.py, nnrpd_access_wrapper.py and
373 nnrpd_dynamic_wrapper.py which should allow you to use your old code
374 without needing to change it.
376 However, before trying to use your old Python code, you may want to
377 consider replacing it entirely with non-Python authentication. (With
378 readers.conf and the regular authenticator and resolver programs, much
379 of what once required Python can be done directly.) Even if the
380 functionality is not available directly, you may wish to write a new
381 authenticator or resolver (which can be done in whatever language you
384 Python Authentication Support for nnrpd
386 Support for authentication via Python is provided in nnrpd by the
387 inclusion of a *python_auth* parameter in a readers.conf auth group.
388 *python_auth* works exactly like the *auth* parameter in readers.conf,
389 except that it calls the script given as argument using the Python hook
390 rather then treating it as an external program. Multiple, mixed use of
391 *python_auth* with other *auth* statements including *perl_auth* is
392 permitted. Each *auth* statement will be tried in the order they appear
393 in the auth group until either one succeeds or all are exhausted.
395 If the processing of readers.conf requires that a *python_auth*
396 statement be used for authentication, Python is loaded (if it has yet to
397 be) and the file given as argument to the *python_auth* parameter is
398 loaded as well (do not include the ".py" extension of this file in the
399 value of *python_auth*). If a Python object with a method "authen_init"
400 is hooked in during the loading of that file, then that method is called
401 immediately after the file is loaded. If no errors have occurred, the
402 method "authenticate" is called. Depending on the NNTP response code
403 returned by "authenticate", the authentication hook either succeeds or
404 fails, after which the processing of the auth group continues as usual.
405 When the connection with the client is closed, the method "authen_close"
406 is called if it exists.
408 Dynamic Generation of Access Groups
410 A Python script may be used to dynamically generate an access group
411 which is then used to determine the access rights of the client. This
412 occurs whenever the *python_access* parameter is specified in an auth
413 group which has successfully matched the client. Only one
414 *python_access* statement is allowed in an auth group. This parameter
415 should not be mixed with a *perl_access* statement in the same auth
418 When a *python_access* parameter is encountered, Python is loaded (if it
419 has yet to be) and the file given as argument is loaded as well (do not
420 include the ".py" extension of this file in the value of
421 *python_access*). If a Python object with a method "access_init" is
422 hooked in during the loading of that file, then that method is called
423 immediately after the file is loaded. If no errors have occurred, the
424 method "access" is called. The dictionary returned by "access" is used
425 to generate an access group that is then used to determine the access
426 rights of the client. When the connection with the client is closed,
427 the method "access_close" is called, if it exists.
429 While you may include the *users* parameter in a dynamically generated
430 access group, some care should be taken (unless your pattern is just "*"
431 which is equivalent to leaving the parameter out). The group created
432 with the values returned from the Python script is the only one
433 considered when nnrpd attempts to find an access group matching the
434 connection. If a *users* parameter is included and it does not match
435 the connection, then the client will be denied access since there are no
436 other access groups which could match the connection.
438 Dynamic Access Control
440 If you need to have access control rules applied immediately without
441 having to restart all the nnrpd processes, you may apply access control
442 on a per newsgroup basis using the Python dynamic hooks (as opposed to
443 readers.conf, which does the same on per user basis). These hooks are
444 activated through the inclusion of the *python_dynamic* parameter in a
445 readers.conf auth group. Only one *python_dynamic* statement is allowed
448 When a *python_dynamic* parameter is encountered, Python is loaded (if
449 it has yet to be) and the file given as argument is loaded as well (do
450 not include the ".py" extension of this file in the value of
451 *python_dynamic*). If a Python object with a method "dynamic_init" is
452 hooked in during the loading of that file, then that method is called
453 immediately after the file is loaded. Every time a reader asks nnrpd to
454 read or post an article, the Python method "dynamic" is invoked before
455 proceeding with the requested operation. Based on the value returned by
456 "dynamic", the operation is either permitted or denied. When the
457 connection with the client is closed, the method "access_close" is
460 Writing a Python nnrpd Authentication Module
462 You need to create a nnrpd_auth.py module in INN's filter directory (see
463 the *pathfilter* setting in inn.conf) where you should define a class
464 holding certain methods depending on which hooks you want to use.
466 Note that you will have to use different Python scripts for
467 authentication and access: the values of *python_auth*, *python_access*
468 and *python_dynamic* have to be distinct for your scripts to work.
470 The following methods are known to nnrpd:
473 Not explicitly called by nnrpd, but will run whenever the auth
474 module is loaded. Use this method to initialize any general
475 variables or open a common database connection. This method may be
479 Initialization function specific to authentication. This method may
482 authenticate(*self*, *attributes*)
483 Called when a *python_auth* statement is reached in the processing
484 of readers.conf. Connection attributes are passed in the
485 *attributes* dictionary. Returns a response code, an error string,
486 and an optional string to be used in place of the client-supplied
487 username (both for logging and for matching the connection with an
491 This method is invoked on nnrpd termination. You can use it to save
492 state information or close a database connection. This method may
496 Initialization function specific to generation of an access group.
497 This method may be omitted.
499 access(*self*, *attributes*)
500 Called when a *python_access* statement is reached in the processing
501 of readers.conf. Connection attributes are passed in the
502 *attributes* dictionary. Returns a dictionary of values
503 representing statements to be included in an access group.
506 This method is invoked on nnrpd termination. You can use it to save
507 state information or close a database connection. This method may
511 Initialization function specific to dynamic access control. This
512 method may be omitted.
514 dynamic(*self*, *attributes*)
515 Called when a client requests a newsgroup, an article or attempts to
516 post. Connection attributes are passed in the *attributes*
517 dictionary. Returns "None" to grant access, or a non-empty string
518 (which will be reported back to the client) otherwise.
520 dynamic_close(*self*)
521 This method is invoked on nnrpd termination. You can use it to save
522 state information or close a database connection. This method may
525 The *attributes* Dictionary
527 The keys and associated values of the *attributes* dictionary are
531 "read" or "post" values specify the authentication type; only valid
532 for the "dynamic" method.
535 It is the resolved hostname (or IP address if resolution fails) of
536 the connected reader.
539 The IP address of the connected reader.
542 The port of the connected reader.
545 The hostname of the local endpoint of the NNTP connection.
548 The IP address of the local endpoint of the NNTP connection.
551 The port of the local endpoint of the NNTP connection.
554 The username as passed with AUTHINFO command, or "None" if not
558 The password as passed with AUTHINFO command, or "None" if not
562 The name of the newsgroup to which the reader requests read or post
563 access; only valid for the "dynamic" method.
565 All the above values are buffer objects (see the notes above on what
568 How to Use these Methods with nnrpd
570 To register your methods with nnrpd, you need to create an instance of
571 your class, import the built-in nnrpd module, and pass the instance to
572 "nnrpd.set_auth_hook". For example:
575 def authen_init(self):
580 def authenticate(self, attributes):
587 nnrpd.set_auth_hook(myauth)
589 When writing and testing your Python filter, don't be afraid to make use
590 of "try:"/"except:" and the provided "nnrpd.syslog" function. stdout
591 and stderr will be disabled, so your filter will die silently otherwise.
593 Also, remember to try importing your module interactively before loading
594 it, to ensure there are no obvious errors. One typo can ruin your whole
595 filter. A dummy nnrpd.py module is provided to facilitate testing
596 outside the server. It is not actually used by nnrpd but provides the
597 same set of functions as built-in nnrpd module. This stub module may be
598 used when debugging your own module. To test, change into your filter
599 directory and use a command like:
601 python -ic 'import nnrpd, nnrpd_auth'
603 Functions Supplied by the Built-in nnrpd Module
605 Besides "nnrpd.set_auth_hook" used to pass a reference to the instance
606 of authentication and authorization class to nnrpd, the nnrpd built-in
607 module exports the following function:
609 syslog(*level*, *message*)
610 It is intended to be a replacement for a Python native syslog. It
611 works like "INN.syslog", seen above.
613 $Id: hook-python 7926 2008-06-29 08:27:41Z iulius $