Mastering the Internet

[Florian Weimer]

* Richard Clayton:

> Last week I asked some people who would know (and I promised not to say
> who I asked or where we were sitting at the time)...  and they said
> quite clearly that the current consultation is about DPI and not about
> Netflow. viz: fairly advanced capabilities.

Well, DPI is rather ambiguous.  I would call it payload retention.
(You have to keep the original, not just the extracted excerpts,
because when the Googles change their internal (Ajax) APIs, you need
to re-run your protocol extractors.)

> viz: the capability envisaged is of reconstructing the streams of data
> which is flowing back and forth between users and "third parties" (ie:
> hotmail/gmail/facebook/bebo &c) and then extracting "traffic data" from
> within those streams of data (and discarding the rest)

I'm still puzzled why this is necessary.  Surely the Googles complain
with orders to provide such data?!  Why build a competing system,
whose more elaborate capabilities turn useless as soon as the first
major site turns on crypto?

> They consider your login name for hotmail/gmail/facebook/bebo &c to be
> traffic data, along with details of who when and how much you are
> communicating with through these systems. They do not consider whatever
> you say within these systems to be comms data but content.

They need to log message identifiers and movements of messages between
folders (otherwise, why bother? you have to cover the shared account
scenario).  If there's no persistent message identifier in the system,
you also need to track some content-derived message indicator (date
and subject line?).

> Essentially this is equivalent to what they'd get from SMTP logs, but
> there isn't the tedious need to nip over to Mountain View for a copy of
> those -- Google helpfully provides them in an easy-to-parse way

You also want to trace access to messages which never leave the draft
folder and are never sent.  (But the Googles likely keep abundant logs
about those as well.)