New Detection Test - Il test più criticato della storia

[cloutz]

La segnalazione è stata fatta su Wilders:
http://www.wilderssecurity.com/showthread.php?t=256947

Si tratta di un test finanziato dalla Symantec, e portato avanti da un certo Dennis Labs.
Qui si può leggere il report: link


perchè è il test più criticato della storia? leggetelo su wilders, dopo aver visto i risultati e letto il report

Per Symantec il report è tutto rilevato ed eliminato alla perfezione, mentre basta che gli altri av mettessero qualcosa in quarantena per avere esito "compromised". Stesso esito "compromised" anche se gli antivirus davano come risposta "blocked", "neutralized", "denied access" o "deleted" (nonostante in realtà l'infezione fosse stata prevenuta).
O per esempio Panda Cloud viene testato senza connessione, in pratica come testare Prevx 3.0 disconnessi

insomma tante cose carine in grado di far per perdere la faccia in 5 minuti..

[xcdegasp]

mi sfugge l'utilità di questo test.. cioè essendo finanziato non mi aspettavo infatti altri risultati, per me sbagli a usare il termine "criticato" .. qui proprio non c'è nulla da criticare perchè è completamente inconsistente che è l'emblema perfetto di "inutile"

[cloutz]

Originariamente inviato da xcdegasp

mi sfugge l'utilità di questo test.. cioè essendo finanziato non mi aspettavo infatti altri risultati, per me sbagli a usare il termine "criticato" .. qui proprio non c'è nulla da criticare perchè è completamente inconsistente che è l'emblema perfetto di "inutile"

LOL

[@Sirio@]

Se ne discute anche sul forum di Comodo.

https://forums.comodo.com/feedbackco...-t47080.0.html

[Romagnolo1973]

Originariamente inviato da @Sirio@

Se ne discute anche sul forum di Comodo.

https://forums.comodo.com/feedbackco...-t47080.0.html

Se è per questo pure quelli di MRG Malware Research Group hanno fatto un test insensato, peccato che non riesco a vederlo, solita pagina che mi dice che non sono umano neppure se mi camuffo da americano riesco più a entrarci, ma comunque hanno testato dei cloud antivirus (panda, prevx e altri) facendo tasto destro analizza campione
Dimostrano con ciò di non aver capito un fico secco dei prodotti che stavano "testando", quindi tutto il mondo è paese, almeno questi qua sono partiti dalla conclusione ovvero mi paga Norton quindi vince e poi sono andati all'indietro in modo convenevole perchè la conclusione si avverasse, penso che pure quelli di Norton non avranno mai il coraggio di postare questo test perchè rischierebbero il ridicolo

[@Sirio@]

Originariamente inviato da Romagnolo1973

Se è per questo pure quelli di MRG Malware Research Group hanno fatto un test insensato, peccato che non riesco a vederlo, solita pagina che mi dice che non sono umano neppure se mi camuffo da americano riesco più a entrarci, ma comunque hanno testato dei cloud antivirus (panda, prevx e altri) facendo tasto destro analizza campione
Dimostrano con ciò di non aver capito un fico secco dei prodotti che stavano "testando", ...

E dove l'hai letto?

[Romagnolo1973]

Originariamente inviato da @Sirio@

E dove l'hai letto?

Su Wilders qua http://www.wilderssecurity.com/showthread.php?t=257523 però poi alla pagina di test come solito io non ci entro , non sono Umano per loro
Tra le altre cose su wilders si dice che questi del MRG che sono gli unici che testano pure CIS entravano su wilders con molti nick per fare un po' di terrorismo mediatico e li hanno cacciati a pedate ma però non ho trovato il preciso riferimento a questa storia

[@Sirio@]

Grazie bello...

Vado a curiosare.

[@Sirio@]

Ho curiosato

Visto che te e forse qualcun altro non può visualizzare la pag. del test eseguito da MRG, incollo quello che IMO è importante.

La premessa:

The daily testing will be on demand, so we will express the usual caveats concerning this limitation and in order to add another dimension to the testing, will, at regular periods conduct live infection prevention tests using samples missed during on demand scanning

Il post riguardo Prevx:

It has come to our attention that with reference to this test, Prevx have implied in their support forum, that our methodology is incorrect and that we do not understand how Prevx works.

In the post they state:

“We haven't gotten any details regarding the test samples so we honestly have no idea if they actually are malicious”

“Prevx's right-click scanner provides only a very small fraction of the detection/protection which Prevx fully provides”

“Testers trying to test Prevx on-demand are incorrect in their methodology and clearly don't understand how Prevx works”

Whilst we would agree, very few people outside Prevx fully understand how Prevx works, we would suggest we do in fact have a good understanding of the product, having tested it behind the scenes for some time (on demand and live infection tests) and also having performed beta testing for a number of releases.

Prevx has provided MRG with licenses so we can test the product in our VMs and I personally liaise with them regularly to discuss results.

MRG is fully aware of the limitations of the standard right click scan performed by Prevx and understand it does not employ all the detection techniques the product has available and that in order to perform a more thorough test, a full system scan is required.

I spoke with Prevx yesterday via Windows Messenger to discuss the results of the test. During that conversation:

1) We confirmed that we had tested the samples using several complete system scans and had not relied on a right click scan. We even detailed the exact numbers detected with each subsequent full system scan, resulting in the total detected.

2) We sent them 25 randomly selected pieces of malware prevx had missed in the sample so that they could analyse them and have an understanding of its makeup.

3) Prevx confirmed receipt of these and stated they were analysing them and indeed gave us feedback on them.

From this, we can prove that shortly after the test was conducted, Prevx:

1) Knew MRG had not relied on a right click scan to conduct the test.
2) Had received a sample of the test malware.

As a matter of routine, we keep logs of all Messenger, Skype and email communications and are happy to provide a full transcript of the conversation, with timestamps, if Prevx wants to try and deny any of the above took place.

MRG has been careful to make clear in these informal tests that on demand testing has limitations. Nowhere have we stated or implied that on demand tests represents exactly how well an application protects a system –this is why we conduct live execution tests as well.

We have contacted Prevx and stated that in the light of our communication with them yesterday and the various caveats we make in the tests and made clear we feel their statement in their forum is unreasonable and have asked them to amend it. Unfortunately, they have not done this and we have not received any communication from them, therefore, we have been forced to detail the information above in order to prove our case.

Because we feel their actions are unreasonable, we now ask Prevx to disable the licenses they provided to MRG for testing their product. We will purchase our own licenses and test their product and publish the results without any feedback or communication with them.
__________________________________
Malware Research Group
Internet Security & Solutions

Giusto per completezza d'informazione.

[Romagnolo1973]

Originariamente inviato da @Sirio@

Ho curiosato

Visto che te e forse qualcun altro non può visualizzare la pag. del test eseguito da MRG, incollo quello che IMO è importante.

La premessa:



Il post riguardo Prevx:



Giusto per completezza d'informazione.

Grazie ma che è !!! dopo neppure un giorno vogliono risposte, mi sa che quelli di Prevx adesso stanno facendo tipo 3 release al giorno del nuovo prodotto quindi tutto ha un tempo... Ma alla fine prevx che è arrivato 8° su 7
Acidini i ragazzi ehhh madonna peggio di un drago di Comodo (l'animale ehh)
Adesso vedo se su linux e Foxyproxy camuffandomi peggio di un Italiano in Svizzera che sta portando indietro i soldi occulti grazie allo scudo riesco a entrare e me lo studio bene bene

[xcdegasp]

pure io no human..

[Romagnolo1973]

Originariamente inviato da xcdegasp

pure io no human..

io ci sono riuscito dopo essermi travestito peggio che a carnevale, poi da Canadese ce la ho fatta, misteri !!!
Allora non è un test dettagliato e quindi lascia dei dubbi, i sample o almeno la tipologia di sample non è specificata se non che sono nuovi nuovi, adesso posto il tutto qui (anche se siamo partiti con un altro assunto ma ormai che ci siamo) così si può leggere liberamente, è un post nel loro forum con questo "test" non ben spiegato nei dettagli, loro stessi lo chiamano "informale", metto anche i loro post successivi:
Cloud Anti-Malware Test using Early Life Samples

MRG will shortly be starting an ongoing testing programme for a cohort of ten anti malware products. Every day the applications will be tested against 250 random samples which emerged within the preceding 24 hours. Each day, a new batch of 250 samples will be used to test the applications, along with the samples from the preceding two days.

The cohort will be composed of five cloud based and five traditional anti malware products. The purpose of this testing programme is to give an overview of which products detect new threats most effectively and to compare the different speed at which the products develop detection for early life samples.

The daily testing will be on demand, so we will express the usual caveats concerning this limitation and in order to add another dimension to the testing, will, at regular periods conduct live infection prevention tests using samples missed during on demand scanning.

As a precursor to the above programme, we thought we would put the five cloud based products through an informal on demand test.

The products tested were:

• Bluepoint Security
• Hitman Pro
• Immunet Protect
• Panda Cloud
• Prevx

The test was conducted on 04/11/09 and made use of the most current versions of the applications at that time. The malware sample used was 500 random samples which were new and received within 24 hours of the test being conducted.

The testing was conducted using fully updated XP Pro SP3 VMs, with live internet connections.

The percentage of malware detected by each application was as follows:

1) Hitman Pro = 66%
2) Bluepoint Security = 60%
3) Prevx = 29%
4) Panda Cloud = 15%*
5) Immunet Protect = 7%*

Please note, applications with * are beta products and this should be taken in to consideration when comparing results.

To give some point of comparison, we tested Microsoft Security Essentials as well, which detected 24%.

It is important to remember that the samples used were very new and it is to be expected that the majority would be missed in an on demand scan as there can be no behavioural analysis etc, which would allow the applications to improve detection. This said however, we feel Hitman and Bluepoint did very well indeed and we look forward to seeing how they perform in our long term project.

Regards,

Chris
Malware Research Group
Internet Security & Solutions
_______________________________________________________
As an addendum, we just ran the same sample by A-Squared on a 22 hour old VM, with no internet connection (to ensure the signatures were from the same time as the cloud apps were tested. A-squared detected 68% of the sample.

As a matter of interest, we will run all these apps against these same samples within the next 3-4 hours to test for improvement.
________________________________________________________
It has come to our attention that with reference to this test, Prevx have implied in their support forum, that our methodology is incorrect and that we do not understand how Prevx works.

In the post they state:

“We haven't gotten any details regarding the test samples so we honestly have no idea if they actually are malicious”

“Prevx's right-click scanner provides only a very small fraction of the detection/protection which Prevx fully provides”

“Testers trying to test Prevx on-demand are incorrect in their methodology and clearly don't understand how Prevx works”

Whilst we would agree, very few people outside Prevx fully understand how Prevx works, we would suggest we do in fact have a good understanding of the product, having tested it behind the scenes for some time (on demand and live infection tests) and also having performed beta testing for a number of releases.

Prevx has provided MRG with licenses so we can test the product in our VMs and I personally liaise with them regularly to discuss results.

MRG is fully aware of the limitations of the standard right click scan performed by Prevx and understand it does not employ all the detection techniques the product has available and that in order to perform a more thorough test, a full system scan is required.

I spoke with Prevx yesterday via Windows Messenger to discuss the results of the test. During that conversation:

1) We confirmed that we had tested the samples using several complete system scans and had not relied on a right click scan. We even detailed the exact numbers detected with each subsequent full system scan, resulting in the total detected.

2) We sent them 25 randomly selected pieces of malware prevx had missed in the sample so that they could analyse them and have an understanding of its makeup.

3) Prevx confirmed receipt of these and stated they were analysing them and indeed gave us feedback on them.

From this, we can prove that shortly after the test was conducted, Prevx:

1) Knew MRG had not relied on a right click scan to conduct the test.
2) Had received a sample of the test malware.

As a matter of routine, we keep logs of all Messenger, Skype and email communications and are happy to provide a full transcript of the conversation, with timestamps, if Prevx wants to try and deny any of the above took place.

MRG has been careful to make clear in these informal tests that on demand testing has limitations. Nowhere have we stated or implied that on demand tests represents exactly how well an application protects a system –this is why we conduct live execution tests as well.

We have contacted Prevx and stated that in the light of our communication with them yesterday and the various caveats we make in the tests and made clear we feel their statement in their forum is unreasonable and have asked them to amend it. Unfortunately, they have not done this and we have not received any communication from them, therefore, we have been forced to detail the information above in order to prove our case.

[gabryflash]

Originariamente inviato da cloutz

La segnalazione è stata fatta su Wilders:
http://www.wilderssecurity.com/showthread.php?t=256947

Si tratta di un test finanziato dalla Symantec, e portato avanti da un certo Dennis Labs.

nel mio campo di ste cose ne vedo tutti i giorni ed è certo piu delicato che il settore pc ...neh xcdegasp?