I did manage some travel, mostly work-related:

• Lerwick, Shetland, for Up Helly Aa. Indeed, my first vacation in some time, and the last I’d have prior to submitting.
• Shanghai.
• Prague.
• San Francisco.
• Abingdon, of course.
• Dublin.

On photography, not so much this year. A minor disaster: I accidentally dropped my shallow depth of field lens (Canon EF 50mm F1.8 II) only to watch it fall into pieces on contact with the floor. Fortunately, the lens itself is fine! But after reassembly, it’s clear the motor unit has been knackered. Still, some of my favourite shots from the last year or so:

Finally some bloggy introspection: I modified my blog this year. I simplified the CSS for the wordpress theme to reduce it to one left-hand column with links, and one right-hand column for text. Only one blog post is shown at a time (many of my posts are long-form), and comments are only enabled on new posts for a short time before being closed off completely. This tidies this blog considerably, though I’d prefer something less heavyweight than wordpress to run the show. One day, I’ll swap out the back-end.

We are nearing 400,000 prefixes in the IPv4 BGP default-free zone. 398,135 are visible to route-views2 in its latest table dump at time of writing. But the allocated address space is not indicative of actual utilisation, nor is the number of prefixes being held in full tables. When it comes to utilisation, it’s the amount of address space advertised by those prefixes that is more interesting.

The IPv4 address space is primarily used for unicast addressing. IPv4 offers 32 bits per address, allowing 4,294,967,296 (2³²) potential addresses. Some blocks of this space are marked for particular uses and cannot be routed globally. These blocks are:

• 0.0.0.0/8 is for self-identification.
• 10.0.0.0/8 is for private networks.
• 127.0.0.0/8 is for loopback addresses.
• 169.254.0.0/16 is for link local addresses.
• 172.16.0.0/12 is for private networks.
• 192.168.0.0/16 is for private networks.
• 192.0.0.0/24 is for “IANA IPv4 Special Purpose Registry”.
• 192.88.99.0/24 is for 6-to-4 relay anycast addresses.
• 192.0.2.0/24 is reserved for examples in documentation (“TEST-NET-1″).
• 198.51.100.0/24 is reserved for examples in documentation (“TEST-NET-2″).
• 203.0.113.0/24 is reserved for examples in documentation (“TEST-NET-3″).
• 224.0.0.0/4 is for multicast.
• 240.0.0.0/4 is listed as for “Future Use”.

That accounts for 588,383,488 addresses that are not globally routeable IPv4 space, leaving 3,706,583,808 from the total.

These are covered in more detail in RFC 5735. Get that “Future Use” block out of your mind right now; those sixteen /8s sure would be useful for a short time, but they’re effectively out of bounds. We don’t know how much hardware explicitly drops packets purporting to have come from, or drops packets destined for, this range.

To reach 100% utilitisation, these ~3.7 billion addresses would all have to be lit up, but we’re not quite there yet. Counting only the shortest prefixes advertised today that fully cover the advertised address space, there are 2,497,743,276 publicly routeable IPv4 addresses out there. (To be clear, since multiple small blocks from a parent block can be advertised over BGP, I count only the shortest prefixes and ignore the overlapping space beneath. Also, I’m simplifying and not discounting special addresses such as broadcast addresses.)

That is, today we’re at about 67.4% utilisation. The Internet is two-thirds full. The last time I checked this, perhaps 2 years ago, we were closer to 50%. I should draw a graph.

Of course, I should make clear that I am not suggesting we should be aiming for full utilisation. Equally, “utilisation” here doesn’t indicate actual use; much of the space advertised is possibly unused, suggesting more efficient packing could take place. But it’s certainly utilised in the sense that it cannot be used by anybody else.

So, I made one: https://github.com/sdstrowes/Glasgow-Thesis-Template

Share, fix, and feed back please!

Broad stats, for interest:

• Number of SVN revisions: 682.
• The dissertation builds out to 163 sheets of A4. Of those:
  • 11 are front matter, including cover, acknowledgements, dedication, abstract, and tables of contents, figures and tables.
  • 119 pages are body chapters.
  • 21 pages are appendices.
  • 12 pages are references.
• I included 130 references.
• Across the body and the appendices, I wrote 40,342 words.

Although word and page counts are not at all barometers of the quality of a PhD dissertation, these counts put me right in the middle of the University’s guidelines.

My word count grew initially in bursts. In the final days, it was hovering around the 41,000 word mark:

The latter chapters were being expanded while earlier chapters were being tweaked down to size, and so the number of words written per day is actually higher than the total words, but I don’t have an easy way to pull out that data. The long flat section between July and August was when work was diverted to work on an INFOCOM 2012 submission.

But the most important meta metric is surely caffeine, which I started logging at the same time I rigged my scripts to log my word count. Broken very simply down into tea and coffee, my cumulative intake is as follows:

That is, 407 mugs logged (230 of which were coffee, 177 of which were tea) across 139 days. Those are the ones I remembered to log, of course. I actually thought my tea count would have been much less than it was, but no matter.

Broken down by week, I think it’s safe to wave my hands in the air and state that there’s a distinct trend toward moar caffeine in the final weeks:

So, there we go. I was burning through a lot of coffee in the final stages while clearing up a pile of annoying consistency glitches. I haven’t read back any of my dissertation since submission, but I think I am happy with the content therein, and the thrust of the argument. I think I have done some good work, and I am pretty happy with it.

Naturally, I have a list of pet issues and errors, without even reading the text. However, turning caffeine into words has now ended. The next phase is the viva, where less caffeine is probably better.

Consider two remote actors, a source sending messages to a sink. If the source is alive prior to the sink, then the source can still send (asynchronous) messages. When the sink is started, the source will finally be able to connect, and messages will be passed. Personally, I think it would be entirely reasonable for messages prior to the sink’s startup to be dropped. Scala’s Remote Actors queues the messages instead.

If messages are to be queued, however, I would fully expect those messages to arrive at the sink in FIFO order. What actually happens is that they arrive in LIFO order, i.e., entirely in reverse. This goes against my assumptions as a developer, given the behaviour of Scala message passing in all other circumstances.

The reason the messages arrive in reverse order is quite simple: the buffer is a Scala list, constructed by attaching each new item as the head of the list. When the sink comes alive, the TcpService running deep within the source is able to successfully connect(), and the message that prompted the connect() is sent. Then, the buffer is drained, by iteratively sending the head of the list until it is empty. Efficient, but surely backwards.

The fix is simple: when a connect() succeeds, reverse and then drain the buffer, and then send the message that prompted the successful connect. Thus, messages are buffered and arrive in FIFO order.

It’s a trivial little behavioural bug, but it could catch somebody out. The fix is simple; I submitted a bug report and a patch.

This post is most closely related to my other posts on Remote Actors: [1] [2].

The opinion survey is protected under copyright law, so apparently it would be deeply uncool for me to start sharing it around on a public website. But I can sure talk about what’s in there.

The opinion survey asked 80 school pupils, 40 parents, 40 early-year University students, and 18 part-time students for their opinion on University funding. Let me quote the report up-front, so that we’re all clear on something:

The findings in this research are based on a qualitative sample which is large enough to provide in-depth indicative findings. However, the findings are not statistically robust and should not be treated as such.

The interviews took place at various locations, and sought to include people from different backgrounds, but we still have the fact that this survey was the only real research that took place to feed into a report that influenced the wrong Government’s policy. (The Browne review was initiated by the previous government.) It’s a survey almost 50% made up of latter-year school pupils, which has been interpreted incorrectly by the Government. Face decidedly in palm.

The basic jist of the report is that questions were asked regarding what levels of fees, if they were to be raised, would be acceptable. The survey also highlighted what I think many of us in the sector implicitly knew: most people are unaware of how much funding the Government provides the tertiary teaching sector per-head.

So the survey has a feel around to find out what people think, and concludes that some people are comfortable with fees set to £6,000 per annum, assuming that £6,000 is the cap. The new £9,000 cap is not mentioned once in the report. Nor is the concept of the withdrawal of the state from tertiary learning in England and Wales, which is the real reason the fees are now rising.

A couple of key quotes from the survey document:

Most full time students and parents believed that the cost of providing higher education should be shared amongst government, graduates and parents.

And also:

Most participants believed that the government contribution to the costs of tuition and support for students is important for a number of reasons: there was the perception that the government benefits from higher taxes from graduates throughout their lifetime and a more skilled workforce able to generate wealth and deliver better public services. Some students and parents believed that government contributions to higher education are important in making it accessible to students from a wide range of different backgrounds, and avoiding a system based on the ability to pay. When completing a ‘pie chart’ exercise, marking relative proportions that the government vs. graduates should pay towards higher education, most full time students and parents believed that the government should pay at least half the cost of higher education. [[Footnote: It is important to note that this exercise did not focus on actual amounts of money, rather on the principle of the burden of funding.]] This is because the personal benefits of higher education were seen by many to match the benefits to society.

So the coalition Government found found this opinion survey that says students and parents might be just happy enough to pay £6,000 tuition per year, but ignored all the parts like this:

most full time students and parents believed that the government should pay at least half the cost of higher education.

And so it seems they worked backward from the amount of money people might be willing to pay to rather ham-fistedly reduce tuition spending, rather than use any additional income to boost tuition spending. If the purpose of the survey was to determine how education spending could be boosted with additional income from graduates, then the results from the survey has been taken and used; the purpose of the survey, on the other hand, seems to have been ignored.

Quite simply, this is a survey that makes the assertion that students will find increased fees acceptable, given deferred payment, but makes no assertions regarding a corresponding reduction in state support. It raises concerns over an emerging “two-tier” system if variable fees are allowed, but does not resolve them. It is fully an opinion survey to determine just how much pain the public might be willing to take, but primarily asks school pupils for the answers. In brief, it is a survey which states its results are not statistically significant given the sample size, which discusses only one half of the course of action taken by the coalition Government.

Update, 2011-03-27: The document released as part of my request is now available through whatdotheyknow.com.

IPv0: Reserved.
IPv1: Reserved.
IPv2: Unassigned.
IPv3: Unassigned.
IPv4: defined in RFC 791.
IPv5: ST-II, defined in RFC 1190.
IPv6: Formerly SIP, then SIPP, defined in RFC 1752.
IPv7: TP/IX, defined in RFC 1475.
IPv8: PIP, defined in RFC 1621.
IPv9: TUBA, defined in RFC 1347.

Certainly the last 4 in this list were designed in response to the IPng white paper solicitation. Jon Postel, for the sake of fairness, applied a version number for each of the candidates, and ultimately IPv6 was chosen.

Regarding IPv{2,3}, the (very informative) internet history mailing list provides an answer: This mailing list post from Vint states that TCP and IP were split sometime around TCPv3, presumably leading to IP packets with version number 3. The suggestion is that there was no IPv2 as we would recognise it.

IPv{0,1} were ever presumably also never used. And just as development of TCP/IP doesn’t map well onto early internet protocol versions, the ‘missing’ versions also don’t map onto NCP, which TCP/IP replaced in early 1983; the packet headers, and indeed the network design and semantics as specified in BBN Report 1822, are very different to what we have today.

Anyway, we squeezed 28 years and 1 month out of IPv4. We’re all well into our IPv6 transition, right?

So, the majority of my reflection focusses on work. I’ve done a lot of work this year, including authoring four papers (primary on two, secondary on two). One of those papers was published at IMC 2010. One was rejected, despite decent reviews, but I did submit it to a highly competitive venue. Two are awaiting a decision some time soon. I find writing fun, in the sense that it features the important process of discovering the things you don’t understand in quite enough detail. This process forces any author to do more work, to clarify their argument. The outcome, I think, is good. And, in my case, the outcome is also two dissertation chapters.

My workflow, I am realising, seems to be heavily centralised, but centralised in various places. By that, I mean, I’m increasingly reliant on parts of “the cloud”. Although my office machine is the focal point for my work, Google has seen some more of my data this year, too. But I’ve expanded into all sorts of other services.

Although I have parts of my electronic life backed up using various mechanisms where appropriate (svn, rsync, etc), these are all a bit disparate. To add to the ragbag, I started using Dropbox as a convenient way of syncing my library of PDF copies of papers between my machines. (I did toy with using lsync to achieve the same, but ultimately left this for another day: I’d like  my own dedicated host sitting in my flat to coordinate this sort of data, but I don’t have one I’m happy with right now.) Regarding other services, I started using Delicious again until all the ambiguity surrounding its future, after which I promptly jumped to pinboard.in, which I like a lot. Another service I’m now happily using regularly is Instapaper, useful for punting links to for reading later, rather than holding far too many browser tabs open. I’ve also been using Todoist for some longer-term todo-list management. All these services feel lightweight, and offer plugins for Chrome, meaning I don’t need to go out of my way to use them. Joy.

In a non-cloud progression, I started learning little bits of R this year. R is a statistics package which makes my life simpler by implementing all sorts of statistical tests that I no longer need to craft by hand: think percentiles and standard deviations, and other calculations of that ilk, and how tedious many of them are to craft in AWK. I let R handle all that now, and my work is probably less error-prone because of it.

Regarding non-work cloudification, Spotify now looms large on my listening habits. I think I started using the service more than a year ago, but I have one machine dedicated to Spotify that is happily hooked up to my sound system and streams much music when I please. Is this good? Well, it is until you decide to do research on Spotify peering traffic using your own account, and you have days where you can’t interrupt experiments. This did make me realise just how much I use Spotify: When I didn’t have access to the service, there were many things I did not own that I could no longer listen to. Equally, there are many CDs I own that I had not heard in many months: physical media still seems more interesting to browse. Will I stop buying CDs? Unlikely, especially given the frequency at which Spotify’s contracts with the record labels seem to expire: It’s not uncommon to find tracks missing from carefully crafted playlists when the track is no longer available through Spotify. A shame, but otherwise I’m pretty happy welcoming Spotify into my home. The other web service I’ve come to use frequently is Dailymile, which I use every single time I go running. It’s fun to see just how my running varies over the year…

In terms of gadgets: I acquired my N900, and my Kindle. These sit very nicely within my ecosystem. In particular, the N900 is basically a Debian-based Linux box. Since I’ve been using Debian systems for many, many years, I’m entirely used to how the N900 works and, as a Linux box, it does a lot of things transparently in my environment. Likewise, it’ll do a lot of things not so transparently, but it pretty much works how I’d like it to. And my Kindle, I’ve only just acquired, but Amazon seem to have been pretty smart in how they manage syncing, and how they manage profile management.

Regarding travel, I haven’t done so much this year. I went to Australia again, with the tiniest of stops in London. I visited Cambridge a few times (arguably too many, given the size of the city). Abingdon, once again, for the annual Cosener’s pilgrimage. And apparently a tour of Scottish cities: Inverness, Aberdeen, Stirling, Edinburgh.

I did also take something of a break at the end of summer to walk the West Highland Way, a 96 mile (154km) walk from Milngavie to Fort William, though I did do this in two stages with a couple of days break in between (the first 48 miles to Crianlarich achieved in two days, then a two day break for aching feet, then I completed the route myself over another three days).

And that other hobby, photography, which I played around with whenever I had time. My favourite shots of this year:

And so, that’s that. Goodbye 2010; hello 2011.

Your research is anchored to one primary requirement: to contribute something to the scientific community. Obviously the primary drive at the start is to gobsmack the world with your sheer brilliance, to make your mark on science, to discover some new fundamental truth, and graduate in record time. Everybody’s entitled to that phase; after all, you never know.

Meanwhile, back here on Earth, we know this isn’t how most Ph.D.s go. “A contribution” can be surprisingly small, often depressingly so. But over time you come to realise that the requirement of contribution is anchored to two important desires, much more important than “fame”: first, the desire to contribute in a way that you hope will be acceptable to your review panel; second, the desire to contribute something that you find interesting. This is, after all, presumably why you got into things in the first place. The former is most important if you care for an easy life; the latter is important if you care for personal fulfilment.

And here is why: you’re going to be working on this for roughly four years of your life. A Ph.D. is one of life’s few opportunities to work relatively freely from interference. Also, if you get to the end of your degree and you still enjoy what you were working on, or have a bag full of ideas for “when you have time,” then congratulations. That’s the primary indicator of whether you were born for your field, or if your Ph.D. is as far as you want to take your academic career. If you get to the end and you aren’t horribly bitter and twisted, itching to get out, then the academic community might decide to keep you around.

So, interest is a primary driver. This interest drives you to learn, and by continuously learning you ultimately also learn where the boundary of human knowledge lies. From the perspective of the review panel, demonstrating crystal clear perception of how far human knowledge goes in your particular field is of the utmost importance.

But there’s a hidden process in getting to this point, and I find it quite interesting.

When you enter from an undergraduate perspective, you think you might know everything about anything, but that’s generally far from the truth. You know a lot, granted, or you wouldn’t have gotten funding. But what many undergraduates have is a broad-strokes picture of things. This is a blissful, innocent phase, where if you want to know more you go read about it.

In this context, everything in the universe feels knowable, because you haven’t yet reached the boundary of human knowledge. The Ph.D. must break you out of this mentality. You will not pass if you continue to think this way. If you do not find the boundary, then you will never be broken.

On your way toward your Ph.D. (which you’ll surely get because you’re pretty smart, right?), you read. You read a lot. You read hundreds of papers every year. Then you read some more. Reading papers is interesting because, apart from a rare few, many cover tiny incremental steps representing one more chunk of the puzzle. Many you can discard immediately. You learn how to rip out the key points of anything you read quickly, to determine in a few minutes whether this piece of paper in front of you that somebody poured months of their life into answers any of your questions. During this time, you refine your questions, and repeat. Refinement followed by reading followed by refinement followed by reading. This phase involves constantly banging your head off the boundary of human knowledge, sometimes without realising it. Still used to answers being available, this is a confusing phase. “Surely somebody did this before? Seems obvious to me.”

During this phase, you hopefully find a crack in the boundary and start working on something without being told to by your supervision team. This is something you decide to fill in yourself, and realise that possibly you could write about it too. Something you can do so that we know just a little bit more than we did yesterday.

But ultimately, the process of the Ph.D. is to break you down and rebuild you into a being who does not expect that certain knowledge is already known, does not feel that somebody must have already found such answers, to arrive at a point where you have an intuitive sense of what has and has not already been achieved. The Ph.D. teaches you how to parse what little information is known in such a way that you can determine whether you need to discover the unknowable. Or prove the as-yet unknown. Or verify the assumptions of others. You earn, through hard labour, sharp criticality of other people’s work. And hopefully of your own, too.

So you start the Ph.D. with much to learn. You finish the Ph.D. with an acute awareness of what is unknown. This, to me, is the single most important point of the whole process. Your Ph.D. becomes your license to go fill in those blanks.

As an aside, scepticism helps the break/rebuild process. The sceptic does not need to be broken down quite so far. Every time the sceptic hits the wall, shrug, “Figures.” The sceptic is quite comfortable in academia, because the sceptic works on the glass-half-empty principle. The sceptic has the in-built ability to detect when he’s read a half-truth. He understands how to interpret words in the least positive, but still truthful, manner, and can determine from that what the author of the text is not saying. This is an important skill not only when scanning the literature for prior work in an area, but also when reviewing papers. Ultimately, in the breaking process, the sceptic reads to the boundary of knowledge, and is capable of realising that nobody knows what the sceptic wants to know.

Shrug. Figures.

But the sceptic is equally pleased when he does find what he’s looking for. The sceptic is content to forever scrape boundary of knowledge, lifted by occasional moments of joy when he finds the information he seeks. The sceptic’s playground is this knowledge boundary.

I am something of a sceptic. I feel that this has made me amenable to much of the Ph.D. process.

The notes that follow are a mixture of what each speaker said, or bullets listed on slides (verbatim, with minor US->UK deltas because I was touch-typing), or thoughts of my own. If you were at IMC and you spot errors (likely), do feel free to point them out, and I’ll fix them.

The amount of text written for each is proportional to certain variables: How tired I was during the talk, how interested I was in the talk, how much I knew about the subject material, and how good the speaker was. Note also that these are pretty raw. Questions are omitted if the question was not clear; many responses from the speaker have been shortened for brevity.

Session 11: Methodology III

Long paper: Revisiting the Case for a Minimalist Approach for Network Flow Monitoring

Speaker: Vyas Sekar

• Decouple collection and computation
  • Generic collection primitive
  • Generic data structure
• Primitive 2: Sample and hold; If flow is already logged, update
• Sample packet with prob p
• If new flow -> create counter
• Use cSamp [NSDI'08] to configure flow sampling capabilities
• Hash-based coordination for non-overlapping sets of flows
• Network-wide optimisation -> operators golas e.g., per-path guarantee
• Putting the pieces together: “minimalist” approach
• Flow sampling + sample & hold
• Relative accuracy difference = accuracy (minimalist) – accuracy (app specific) / accuracy (app specific)
• Hardware reqs are similar:
  • both need per-packet array/key-value updates
  • More than packet sampling, but with router capabilities
• Memory for counteres
  • botleneck is SRAM (flow headers can be offloaded to DRAM)
  • We conservatively assume 4x more per-counter cost
• Processing costs
  • Online cost lower for minimalist (don’t need per-app-instance)
  • OFfice cost is higher for minimalist (but can be reduced)
• Reporting bandwidth: Higher for minimalist, but < 1% of network capacity

• Q: Costs
• A: Hardware costs. Full sampling is of similar complexity to current packet sampling. Another issues is SRAM size.

Long paper: Exact Temporal Characterization of 10 Gbps Optical Wide-Area Network

Speaker: Daniel A. Freedman

• Understand novel behaviour of high-performance lightly loaded WAN links
• Apprciate distortive impact of endpoint network adapters
• Present instrumentation (named bifocals)
• Network adapter distorts signatures/timestamps. Instead, let’s tap optical fibre directly.
• Record analogue waveform of entire physical layer symbolstream in real time; extract discrete ethernet frames in later off-line processing
• Packet chains are exponentially rare
• What causes WAN burstiness?
  • Still an open question
  • But… Relatively insenstive to background traffic. Not due to a single router
• Instrumentation: Precise and reproducable timing measurements, accessible and transparent
• This is the first “ground truth”measurement of packet timings of network strreams in flight
• Comparaitive refernce for less accurate tools (determine when/where applicable)

• Q: If we had optical switches, would this behaviour still be there?
• A: That’s two questions. Is it the conversion? I think no. (“ODE”?). Is it the design of the switches? I guess so. Doesn’t know enough about optical switches though. If the switches mimic the same architecture, then probably they’ll behave the same.
• Q: You get a number of time peaks at different points. Did you find any time peaks that would correspnd to timings within the router device? Known clock frequencies, etc? Could start talking about where in the router these artifacts originate?
• A: Haven’t seen precisely what you’re asking about.

Long paper: TraceNET: An Internet Topology Data Collector

Speaker: Kamil Sarac

• tracenet is an extension to traceroute in internet topology collection studies
• At each hop, traceroute to nodes in subnet hosting that hop’s ip addr
• Network exploration
  • Trace collection
  • subnet exploration: start at a /30 subnet S which inclueds the IP address IP_v of the pivot interface v; Check if /30 mate of IP_v is likely to be in S; Grow S to a /X subnet (X = 29, 28…) which includes the IP address of v as long as IP addresses in subnet S satisfy our hueristics
• Hueristics: 9 heuristics to improive accuracy of our inferences
• Does IP address i belong to assumed subnet S? Yes with high probability, if it satisfies hueristics.
• Probing overhead: Exact number of probes during a tracenet session depends on length of trace, configuration of the explored subnet, confiruation of the fringe subnets
• Path fluctuations
• LImitations: Depends on heuristics based on best practices in layer 3 network configuration underestimation due to subnets with sparsely utilised IP addresses requires router responsiveness

• Q: How sure are you that the 9 heuristics cover all the cases?
• A: We did our best to think of all cases! Blah blah blah.
• Q: Did you get multiple responses from broadcast addresses?
• A: We did not do it.

Session 12: Online Social Networks

Long paper: Understanding Latent Interactions in Online Social Networks

Speaker: Jing Jiang

• Crawling process complicated by two things:
  • crawlers must revisit usrs often to build complete log; new visitors cause older visitors to fall off the list.
  • crawlers cannot visit each user to frequently; renren imposes multiple rate limits
• Visible interactions show much reciprocity

Short paper: Measuring the Mixing Time of Social Graphs

Speaker: Abedelaziz Mohaisen

• The Sybil Attack, John Douceur
• Social graphs are not fast mixing
• Some theoretical arguments in sybil defences are inaccurate
• the applicability of social network-based sybil defenes is infeasible for some social graphs
• there is a negative correlation between trust (hypothesized) and the underlying algorithmic property in these graphs—i.e., the mixing time. (Thanks to Aziz for correcting my hasty notes!)

Long paper: Estimating and Sampling Graphs with Multidimensional Random Walks

Speaker: Bruno Ribeiro

• Measurement distortions

Short paper: The Impact of YouTube Recommendation System on Video Views

Speaker: Lixin Gao

Session 13: Path Properties and Dynamics

Long paper: Selecting Representative IP Addresses for Internet Topology Studies

Speaker: Xun Fan

• Topology study needs targets. Where do you dervice your destination addresses for your traceroutes?
• census (ping sweeps) tell us who responds. Use census to predict best representatives.
• contribs:
  • new automated hitlist generation method, updated everyh 2-3 months
  • evaluation of the hitlist
  • learn about the internet from the hitlist
• Completeness: Select one representative for all allocated /24′s Responsiveness: representative should be most likely to respond. Stability: representatives don’t change arbitrarily. Important for longitudinal study.
• Representative predicition from census data: score IP addresses based on appearances in census data. Choose highest scoring rep per /24.
• Dataset available: http://www.isi.edu/ant/.

• Q: Your aim is not to map the core. But have you considered in fact doing this?
• A: ???
• Q: What is the best coverage both in terms of ASes and prefixes that you can get? How much can I probe?
• A: 1/12th of the allocated address space

Short paper: Representative Speed Tests for Free: Estimating Achievable Download Speed from Passive Measurements

Speaker: Jeffrey Pang

• How fast is the network? (From the perspective of the customers)
• Benefits of passive measurement: No additional load on network,
• millions of vantage points, captures actual “user experience”
• Challenge: estimating network speed from flow records in the wild.
• Solution: Only look at flows from non-rate-liimted applications and content-providers.
• Goal: Detect rate-limited flows. Detecting if individual flows are rate-limited is not possible since we only have record flow records, not full packet streams.
• Goal: Detect rate-limited flows-types
• Contribution: A method to generate a throughput index. A set of creiteria that selects on y passive measurements that accuraely reflect achievable network speed

• Q: Why didn’t you consider normal TCP behaviour?
• A: Because at the flow level we do not have that information, do not know RTTs, etc.

Long paper: YouTube Traffic Dynamics and Its Interplay with a Tier-1 ISP: An ISP Perspective

Speaker: Vijay Kumar Adhikari

• Q: If you are doing localisation, you would see large sites near to customers?
• A: Some faff on cost of building, economics, hand-waving. ??

Short paper: On the Characteristics and Reasons of Long-lived Internet Flows

Speaker: Lin Quan

• Long flows -> high probability these are computer-to-computer traffic
• Contrib: Multi-time-scale analysis
  • New mechanism that csales to weeks of traffic
  • efficient processing and queries
• show existence of long-lived flows
• Find their characteristics and causes
• Causes:
  • port based analysis
• http://www.isi.edu/ant/

• Q: What’s the smallest timescale you can say anything about the bustiness?
• A: 10 mins
• Q: Okay, 10 minutes is not enough to say anything about the burstiness
• Q: How about using 2-tuple rather than 5-tuple; src-ip:dst-ip?
• A: Um.

Session 14: Anomaly and Event Detection

Long paper: BasisDetect : A Model-based Network Event Detection Framework

Speaker: Paul Barford

• Anomoly detection: DOS attacks, outages, flash crowds, etc
• Goal: Fast and accurate identification
• Challenges: Data selection, filter selection,, perspective, thresholds, etc
• Little actual deployment and use
• Can we exploit structure in both normal and anomalous traffic to identify anomalies?
• Basic detect methodology. Three components:
  • anomoly dictionary construction
  • anomoly decomposition of link data
  • network-wide data fusion
• In terms of single-link real world dataset: over 48% frewer false alarms
• in terms of network-wide synthetic dataset: between 54-90% fewer false alarms

• Q: If I understand, you’re looking for anomolies in the future that look like the old anomolies. Relies on signatures, won’t be able to see something you’ve never seen before?
• A: Yes. False negatives are way better than false positives. We’re focussing on the latter, not enough on the former.
• Q: Did you focus at all on false negatives?
• A: A little.
• Q: What about building your dictionary on normal traffic, and assuming anything else is anomolous?
• A: ??

Short paper: Temporally Oblivious Anomaly Detection on Large Networks Using Functional Peers

Speaker: Kevin M. Carter

• Assume baseline behaviour, aim of anomaly detection is to flag the unexpected
• An anomaly is not necessarily malicious
• Given a large network, utilise functional peers to identify anomalous hosts. Baseline over space rather than time.
• Contribs:
  • remove temporality entirely from detection
  • scales to very-large networks, don’t have to maintain state for all hosts
  • anomalies can be automatically categorised into descriptive sets. Actionable intelligence.
• This is NOT an IDS.

• No questions fielded!

Long paper: What Happened in my Network? Mining Network Events from Router Syslogs

Speaker: Dan Pei

• Contrib: SyslogDigest: proacitvely mine network events from router systlogs
• take low-level minimally sturcture syslog messages -> prioitised high-level network events.
• Grouping relevant messages together
  • Temporal relationship
  • spatial relationship
  • associations of message types
• Presentation:
  • ranking
  • naming
• Message template learning: Templates trte the signatures with the combination of frequent words
• Decompose messages into works separated by whitespace
• Construct a tree structure to express the template hierarchy based on the input messages
• [...]
• Rule-based grouping
  • “happening together”. Temporaly close within a sliding window
  • “together frequently”. Association rule mining.
• Offers 2-3 orders of magnitude reduction from raw messages to events