Io uso questo... e' codice buttato la', (diciamo pure che e' uno schifo :D), ma il suo lavoro lo fa.
Semplicemente pesca fuori i nuovi thread nelle sezioni indicate (nel caso 7,34,38 ovvero Linux, Programmazione e... boh :confused: ) e te li spedisce per posta all'indirizzo indicato. Devi solo cambiare le prime righe, dove devi specificare dove hai salvato il programma (data_path), il destinatario della posta (recipient) il server tramite cui mandarlo (smtpserver), il mittente (fromadd) e l'elenco dei forum. Lo scheduli che parta ogni qualche ora e sei a posto.
import sys,re,time,urllib,smtplib,urllib2
# Change those
data_path = '/dati/Coding/forum_parser/'
recipient = 'marco_righele@yahoo.it'
smtpserver = 'localhost:25'
fromadd = 'marco_righele@yahoo.it'
forum_list = [ '7' , '34' , '38' ]
verbose_level = 4
def log( level , txt ):
if level < verbose_level:
print txt
def download( uri ):
log(3,'downloading page ' + uri )
x = urllib2.urlopen( uri ).read()
log(3,'done !')
return x
def prepare_html( topic_list ):
body = ("<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 4.01 Transitional//EN\">\n<html><body>"
+ "<p>The following topics appeared on the Forum:</p>\n\n" )
for m in topic_list:
(link,title) = m
body = body + "<p><a href=\"" + forum + link + title +"</a></p>\n"
body = body + "</body></html>"
return (body,'text/html')
class forum_parser:
def __init__( self , name , uri , fetch , forum_list , regex , num_regex , seen_name ):
'Subject of the mail'
self.name = name
'Base address of the forum'
self.uri = uri
'Fetch page base uri'
self.fetch = fetch
'Forum numbers to be appended to the fetch page'
self.forum_list = forum_list
'Regular expression used to find the link and the title'
'it needs two groups, the first to get the (relative) link, the second to get the topic title'
self.p = re.compile(regex,re.IGNORECASE|re.DOTALL )
'Regular expression used to find the topic number from the thread link'
self.find_num = re.compile(num_regex)
self.seen_file = seen_name
'Has the topic to appear or not in the seen list ?'
self.must_appear = False
self.load( data_path + seen_name )
def load(self,file_name):
log ( 1 , 'loading ' + file_name )
self.seen = {}
try:
c = file(file_name).readlines()
for i in c :
self.seen[ i.strip() ] = True
except IOError:
self.seen = {}
except ValueError:
log(1,'Warning: invalid data found in seen_file: using default values')
self.seen = {}
def save(self):
f = file(data_path + self.seen_file,'w')
for i in self.seen:
f.write( str(i) + "\n")
def generate( self , topic_list ):
body = "The following topics appeared on " + self.name + ":\n\n"
for m in topic_list:
title = m['title']
link = m['link']
body = body + title + "\n" + self.uri + link + "\n\n\n"
return (body,'text/txt')
def get_num( self , uri ):
return self.find_num.search( uri ).group(1)
def parse( self , contents ):
log(3,'parsing...')
iterator = self.p.finditer( contents )
unknown = []
for match in iterator:
link,title = match.group(1),match.group(2)
thread_num = self.get_num( link )
unknown.append( { 'link':link,'title':title , 'thread_num':thread_num } )
unknown = self.remove_unwanted(unknown)
if len( unknown ) > 0:
return self.generate( unknown )
else:
return ('','')
def remove_unwanted(self,link_list):
'''Filter out unwanted topics, and return a new list'''
log(3,'removing unwanted topics')
result = []
for i in link_list:
log(3,'considering thread ' + i['thread_num'] )
if self.seen.has_key( i['thread_num'] ) == self.must_appear:
log(3,"I'm going to keep it")
result.append( i )
self.seen[ i['thread_num'] ] = True
return result
def load_page(self,page):
p = self.uri + self.fetch + page
try:
return download(p)
except IOError:
log( 0 , "Warning: unable to fetch page " + p )
return ''
def go(self):
page = ''
for p in self.forum_list:
page = page + self.load_page(p)
if page != '':
(body,content_type) = self.parse( page )
if ( body != '' ):
log( 1 , "Sending message to " + recipient )
message = ("To: " + recipient
+ "\nFrom: " + fromadd
+ "\nSubject: " + self.name
+ "\nDate:" + time.strftime("%a, %d %b %Y %H:%M:%S -0000", time.gmtime())
+ "\nContent-type: " + content_type +" ; encoding=utf-8"
+ "\n\n" + body )
serv = smtplib.SMTP(smtpserver)
serv.sendmail(fromadd,recipient,message)
self.save()
else:
log( 0 , "No new topics,exiting..." )
else:
log( 0 , "No pages loaded, connection down ?" )
hw = forum_parser( 'HWUpgrade Forum' ,
"http://forum.hwupgrade.it/",
'forumdisplay.php?s=&forumid=',
forum_list ,
"<a href=\"(showthread.php\?s=[0-9a-zA-Z]*\&threadid=\d+)\"><b>([^<.]*?)</b>" ,
r'threadid=(\d+)'
, 'visti.txt' )
hw.go()
Il risultato ti arriva in posta in posta in formato HTML nella forma seguente:
The following topics appeared on HWUpgrade Forum:
mount pennino usb
http://forum.hwupgrade.it/showthread.php?threadid=742096
Installare un tema di Firefox
http://forum.hwupgrade.it/showthread.php?threadid=740209
usare msn in linux
http://forum.hwupgrade.it/showthread.php?threadid=708711
Tasto Funzione
http://forum.hwupgrade.it/showthread.php?threadid=742246
Lingua Ungherse su MDK 10
http://forum.hwupgrade.it/showthread.php?threadid=742212
[SuSE] Ho perso il bootsplash
http://forum.hwupgrade.it/showthread.php?threadid=742202
svgalib e /dev/vga che non c'è!
http://forum.hwupgrade.it/showthread.php?threadid=742196
modificare un forum PHPBB
http://forum.hwupgrade.it/showthread.php?threadid=742215
MySql da incubo
http://forum.hwupgrade.it/showthread.php?threadid=742142
[java] documentazione javadoc
http://forum.hwupgrade.it/showthread.php?threadid=742172
Il codice potrebbe essere piu' breve in effetti... ma lo uso anche per altri forum e quindi ho dovuto generalizzare un po'.