Ajax

Over in SproutCore land, they have been talking about Peter Bergstrom and his amazing work with Canvas and SVG:

Peter Bergstrom has been doing some amazing work with SVG and canvas tags in his SproutCore-based these project called PaperCube.  PaperCube visualizes citations their relationships between authors.  Watching the videos of his project, you’d swear he was using Flash or Silverlight, but its not.  He’s using only native web technologies powered by SproutCore and JavaScript.  It’s a great example of what’s possible using the browser’s capabilities today.

Checkout PaperCube’s Node Graph (SVG),  Per Year View (SVG), and Paper Tree (pure HTML).

Starting to get goose pimples about 2009 :)

Jacob Seidelin has a really nice 2008 roundup of awesome JavaScript-yness that focuses a little on his own area of expertise (canvas whiz and all):

2008 has been just great, not least because of all the great stuff people have been doing with JavaScript, the new canvas element and the web in general. Browser vendors are now competing, trying to outdo each other in terms of JavaScript performance, leaving developers and end-users as winners as we now get to do cool stuff that wasn’t really feasible before. Here’s a summary of the neat things of the year 2008 as I saw it.

He starts out with fun canvas games such as Super Mario, Pac-Man, Breakout, Space Invaders, Bomberman or T&C Surf Designs.

Then we get to some fun demos… including small ones Rubber effect, Tunnex, Mars.

Then we get to 3D: Everything from basic 3D cubes, texturing experiments to regular JavaScript/Canvas 3D engines have been done.

And, emulators:

That JavaScript performance is getting better and better must be true since it’s now possible to emulate other systems in the browser. Matt Westcott (who also made the Antisocial demo) pulled another rabbit out of the hat, JSSpeccy the ZX Spectrum emulator. That’s just way cool.

James Urquhart also played a bit and created a proof of concept SCUMM interpreter. It won’t let you play Day of the Tentacle or even Maniac Mansion, but supposedly it can at least let you see a bit of OpenQuest.

A personal favorite of mine was António Afonso’s JavaScript AGI interpreter. AGI was the system used by Sierra in the golden days of adventure gaming and when António made this little gem with an (almost) working Leisure Suit Larry, it just totally tickled my fancy.

This is just the start, as we get to audio, visual, and other cool fun times from 2008. Thanks for the great roundup Jacob. I think 2009 will be even better!

Alvaro Videla has a fun post on extending the WebKit Web Inspector with something a little strange.... a game:

Alvaro walks through how to take some JavaScript/CSS/HTML and plug it into the inspector world (normally hidden somewhere like /Applications/WebKit.app/Contents/Frameworks/10.5/WebCore.framework/Versions/A/Resources/inspector) and adding panels (e.g. see the new game panel here:)

PLAIN TEXT JAVASCRIPT:

1.  
2.     this.panels = {
3.         elements: new WebInspector.ElementsPanel(),
4.         resources: new WebInspector.ResourcesPanel(),
5.         scripts: new WebInspector.ScriptsPanel(),
6.         profiles: new WebInspector.ProfilesPanel(),
7.         databases: new WebInspector.DatabasesPanel(),
8.         game: new WebInspector.GamePanel()
9.     };
10.  

Robert Tomsick has been playing with Sandboxed Safari, a project that aims to let you use the Leopard sandbox feature with the browser, via a little launcher:

When Leopard was released, one of its big selling points was its "sandbox" feature. This garnered a fair bit of attention, as sandboxing is a fairly new feature for consumer operating systems.

A lot of people (myself included) assumed that Leopard's sandbox system would be used for Safari, seeing as how web browser exploits are a tremendously popular attack vector. Unfortunately, Leopard's version of Safari is not sandboxed. In fact, very little of the OS actually is. I don't know the real reason behind this, but I suspect it had to do with the release date pressures. Although Leopard ships with a number of sandbox profiles for things like syslog and ntpd, they are not used in the default config. Pretty much the only things that are sandboxed are mDNSResponder and some xgrid daemons. A quick look at the comments in the existing policies indicate that Leopard's sandbox system (named "seatbelt") is rather... buggy. As it turns out, the comments don't lie -- seatbelt *is* quite buggy, at least as of 10.5.6. Still, it's an extraordinarily powerful (not to mention cool) feature, and it's got a lot of potential to increase the security of Mac OS X.

But enough rambling about seatbelt. I'll make a few pages documenting what I've learned about it -- but until I get around to doing that, let's talk about Safari. Basically, I found that Leopard's sandbox system, buggy though it may be, is sufficiently mature as to allow the development of a seatbelt policy for Safari, albeit with some caveats. SandboxedSafari is my attempt at developing such a policy.

He continues to discuss the flaws: No plugin support, overly-permissive process-exec rules, no network filtering, no address book integration, no keychain, focus issues.

It will be interesting to see if Safari 4 + Snow Leopard gets us this out of the box.

At a New Years Eve party, a friend help up a drink and toasted to his company deciding to discontinue direct support of IE 6 in 2009, and letting users know that the site may work better with IE 7 or another latest browser.

Then, Asa Dotzer puts up a chart of the IE 6 numbers:

Still far too high a percentage and enough to make you grown. Also, the last few pounds are the hardest to lose. The good news for me is that on most Web applications that I personally work, and definitely those that I work on in early 2009 will have much different numbers.

Maybe IE 8, Windows 7, and the great new browser war will help, or maybe some percentage is for lost computer souls.

Today we are fortunate to have a guest post by Patrick Lightbody, most recently of BrowserMob fame (and previously Selenium work, OpenQA, WebWork, and more). Let's listen in to him talk to us about load testing, and let him know your thoughts in the comments below:

I've been developing and testing complex web apps for a long time. I was the co-creator of WebWork (now Struts 2.0) and an early champion of DWR, writing one of the first AJAX form validation frameworks for Java web apps. But over the years, I noticed that as our web technologies and techniques got more sophisticated, our testing techniques were not keeping up.

That was why I founded OpenQA and helped grow Selenium to the popular testing tool that it is today. Selenium helps with functional testing of complex AJAX apps, but there isn't an equivalent for load testing, which is why I started BrowserMob, a new type of load testing service.

Traditional load testing

In order to achieve high levels of concurrency, traditional load testing tools (both open source and commercial) work by sending large numbers of HTTP requests as a way to simulate many concurrent users interacting with your web page. These tools work by recording the traffic that comes from a browser session and then requiring that the load tester tweak a generated script so that it worked properly when played back X times concurrently.

Common problems would be that the initial recording would embed in cookie values that were tied to individual sessions. Additional unique state might be encoded in other hidden form elements, all of which required some fine tuning after the fact. If you've ever tried to run a load test, this is probably a very familiar process. It has worked reasonably well up until recent years, but AJAX has made this process even more difficult.

Ajax + load testing = hard

The reason Ajax has complicated things is that it encourages more logic and state to run inside the browser session. This means that just watching the traffic across the wire doesn't necessarily tell the full story. The richer an app gets, the more difficult it gets to simulate the exact effects of hundreds or thousands of users hitting your site.

This is the problem I decided to solve when I started BrowserMob. It's on-demand, low-cost and uses real browsers to completely change the way load testing is recorded and played back.

Do real browsers really matter?

Real browsers absolutely matter. There are two major reasons:

1. It simplifies the script creation process by letting you avoid all the complexities and hacks you have to do with traditional load testing tools.
2. It ensures that you'll see 100% of the traffic and load against your site that a real user would cause.

We'll look in-depth at each of these topics separately to see how use of real browsers helps and how a service like BrowserMob compares to existing load testing technologies.

Simplifies script creation

In today's modern web applications, AJAX is just about everywhere. And we're not necessarily talking about super rich applications like Google Maps or Yahoo Mail, but even simple sites like google.com now use advanced AJAX techniques. See Google's auto-complete for a real-world example:

In this case, when typing values in to the search box, the web browser executes JavaScript logic that in turn makes AJAX calls to Google's search engine, asking for search suggestions to display. It does this on every keystroke that the user types in. This is a standard auto-complete control that most Ajaxian readers are very familiar with.

When recording a script with a traditional load testing tool, one of two things may happen here:

• The recorder will see the AJAX traffic and capture it for playback in the load test
• The record will not see the AJAX traffic and will only capture the request made when the user clicks the "submit" button

Obviously these Ajax requests are causing real load, so we want to make sure they get played back in a load test. Let's assume you're using a tool, such as JMeter, that does capture the AJAX traffic. Here's what that looks like:

The recorded traffic is effectively:

http://clients1.google.com/complete/search?hl=en&gl=us&q=b http://clients1.google.com/complete/search?hl=en&gl=us&q=ba http://clients1.google.com/complete/search?hl=en&gl=us&q=ban http://clients1.google.com/complete/search?hl=en&gl=us&q=bana http://clients1.google.com/complete/search?hl=en&gl=us&q=banan http://clients1.google.com/complete/search?hl=en&gl=us&q=banana

Each key stroke by the user is included in each subsequent search term. Let's ignore the requirement of validating the results that come back from the AJAX requests for the moment (they are usually in JSON or XML format and difficult to validate using most tools). Instead, let's just add a twist to the load test requirement for doing searches: the load test must search from 100 different search terms.

Parameterization is very common requirement, since it ensures that the load is realistic and doesn't get cached in any unnatural way. This means that now in addition to searching for the term "banana", we're also searching for "apple", and "orange", among others.

However, this means your script can't just blindly submit requests to those previous URLs either, since those were tied to the "banana" term. Instead, they must search for the sequential characters of the respective search term, such as:

http://clients1.google.com/complete/search?hl=en&gl=us&q=a http://clients1.google.com/complete/search?hl=en&gl=us&q=ap http://clients1.google.com/complete/search?hl=en&gl=us&q=app http://clients1.google.com/complete/search?hl=en&gl=us&q=appl http://clients1.google.com/complete/search?hl=en&gl=us&q=apple

Unfortunately, this is where even the best traditional load testing tools fall down. They don't provide any help here, so it's up to you to figure out how to, if it's even possible, write complex scripting logic that breaks down the randomly selected search term by characters and then subsequently issue Ajax requests for each character in the term.

At this point, you're basically rewriting the same logic that the web app developer wrote originally. If you're a QA engineer, this may be difficult since you don't know all the internal AJAX logic coded in to the application. If you're the developer, it's still annoying because it's tedious and likely in a language other than the original JavaScript that you wrote your code in.

So how do real browsers help?

Because BrowserMob uses real browsers to both record and playback load, that means you don't have to worry about trying to simulate the logic in a web browser. Instead, all you have to do is record the human interaction with the browser, such as typing in a randomly selected search term. BrowserMob will then pass those instructions on to the hundreds or thousands of browsers participating in the load test, and those browsers will in turn "do the right thing" and issue the proper AJAX requests.

And if the underlying logic, such as the request URL pattern for those AJAX requests, changes? With traditional load testing it's up to you to detect and fix the problem. If your test uses real browsers to play back the traffic, your script won't need to change one bit - the new AJAX logic will be run by the browser in real time.

Ensuring realistic playback

We've seen how use of real browsers helps with script creation, but what about playback? As we just learned, using real browsers simplifies the process of recording and shrinks the behavior coded in to the script itself. This means we're letting the real browser - the same type of program your end users will use - make the decisions about what requests to make.

For example, when visiting http://ebay.com you might see the following page:

But reload the page and now you might see this:

Notice a difference? The upper right section has completely different images displayed. That's because eBay's home page chooses what to display based on complex and multi-variant logic determined at runtime. It's quite likely that it's going to be impossible for a load tester to know which images will be displayed on any given request.

It's true that some load testing tools will try to parse the pages in real time and figure out which images should be displayed, but that's hardly comforting once you've already learned they can't deal with even the most simply Ajax components, as we just saw. And as most AJAX developers know, resources such as images and stylesheets are more and more likely to come from complex JavaScript logic and not due to a simple static reference in an HTML page.

Instead, the only way to guarantee that every single object (image, JavaScript, AJAX request, advertisement from an ad partner, etc) gets requested is to use a real web browser during playback. While it is much more resource intensive, it is also a major time saver on both the front-end, as scripts are much simpler to write, and the back-end, as you can be confident that the most realistic level of load was produced.

So next time you hear of load testing happening on one of your Ajax apps, make sure those doing the testing understand the complexities and difficulties associated with testing a complex web app. Help them be on the lookout for the issues highlighted here.

Thanks to Patrick for writing this. Do you have something important to say? If so, contact us with your idea!

How about if all you needed to do was:

PLAIN TEXT JAVASCRIPT:

1.  
2. var server = new Addressable.Server({ useGears: true });
3.  
4. server.onmessage = function (message) {
5.         log(message)
6. }
7.  
8. server.connect(function (id, url) {
9.     log("Connected. Messages will appear here.")
10.     $("#clientId").html("Client-Url: "+url)
11.     $("#testForm").attr("action", url)
12. })
13.  
14. function log(msg) {
15.     var log = $("#log");
16.         log.html(""+msg + "<br />" + log.html())
17. }
18.  

And you would have Comet end points to play with?

That is what Malte has given us with a proof of concept called UniversalComet:

I have developed a JavaScript library that makes it extremely easy to incorporate "comet" or server-push technology into any web application without need for special programming or any special support on the server. Ressources needed on the server are also practically zero.

When you use the JavaScript library, you will receive an URI that identifies the client. Now you can make simple HTTP-GET requests to that URI with a message parameter and the client will receive the message immediately. Because only your application will know the un-guessable URI, the client should be save from unwanted messages. The use of HTTP-GET messages also allows sending messages in a "P2P"-fashion from client to client using JSONP cross-domain-messages.

The function that is passed to the connect-method receives the URI of the client and as soon as messages arrive they will be send to the onmessage-callback.

The JavaScript application uses "channels" to differentiate between different windows that the same client has open at any given time. These are cached in window.name. If you set window.name yourself this value is used. You can also set server.setChannel("myChannel") before you connect to name the channel yourself.

You can check out the demo.

John Resig wants us to file bug reports to browser vendors but what about accessibility? Is that a responsibility that we have as Web developers?

Todd Kloots of Yahoo! shows us how to configure our machine for screen reader testing with full instructions:

When developing using the WAI-ARIA Roles and States, you need to test your code in a screen reader to ensure everything is working as you expect. As a follow up to my presentation on Developing Accessible Widgets with ARIA and in the interest of helping other developers test their code, I thought I would provide some tips on how to configure your development environment for screen reader testing.

1. Install virtualization software
2. Install browsers & take a snapshot of that state
3. Install and configure screen readers
4. Restart the virtual machine & take a snapshot of that state

The paper on MD5 considered harmful today delivered at the 25th Annual Chaos Communication Congress in Berlin has got people scared again.

The team showed an MD5 collision which is well explained by Simon Willison (he is so good at getting to the meat, a tough skill indeed):

Use an MD5 collision to create two certificates with the same hash, one for a domain you own and another for amazon.com. Get Equifax CA to sign your domain’s certificate using the outdated “MD5 with RSA” signing method. Copy that signature on to your home-made amazon.com certificate to create a fake certificate for Amazon that will be accepted by any browser.

Mozilla Security folks issued an advisory which included the impact to users:

If a user visits an SSL site presenting a fraudulent certificate, there will be no obvious sign of a problem and the connection will appear to be secure. This could result in the user disclosing personal information to the site, believing it to be legitimate. We advise users to exercise caution when interacting with sites that require sensitive information, particularly when using public internet connections.

Status

This is not an attack on a Mozilla product, but we are nevertheless working with affected certificate authorities to ensure that their issuing processes are updated to prevent this threat. Mozilla is not aware of any instances of this attack occurring in the wild.

Microsoft also advised.

Then we get SSL in perspectives which talks us through 2008:

1. Dan Kaminski shook world’s faith in DNS. BTW, you already checked your DNS hardness or switched to OpenDNS, didn’t you? Anyway, DNS security or not, you cannot trust non-SSL traffic when you’re traveling, or you’re behind a proxy you can’t control (TOR, for instance), or otherwise not using a trusted ISP… wait, do you really trust your ISP? OK, you should not trust non-SSL traffic, period.
2. But then, Mike Perry demonstrated how cookies can be stolen from SSL-secured sites (and NoScript deployed some countermeasures).
3. Unfortunately, a shameful incident revealed that you can easily buy a valid SSL certificate for a web site you’re not related with, if you find an unscrupulous enough vendor: in this case, a mozilla.com certificate has been obtained by Eddy Nigg of StartCom Ltd. from the Certstar Comodo reseller, no question asked. Of course, as a work-around, you could remove the offending CA root, but you must expect side effects (I discovered this breaks cleverbridge e-commerce back-ends, for instance). And, most important, are you sure this is the only sloppy CA out there?
4. As if this didn’t suck enough, a speech has been given today at 253c by Alex Sotirov, Arjen Lenstra and other high-profile researchers, who managed to leverage known MD5 weaknesses and not-safe-enough practices of some certificate issuers to build their own rogue CA.

It then talks about Perspectives the Firefox plug-in that compares cert hashes with a database of known fingerprints to detect false certs.

Phew. Two Web security pieces almost in a row :/

Thanks to the folks at Adobe, we've got video of pretty much every session from the Ajax Experience 2008 that we can share on-line, free of charge. We'll release them over the next week or so in batches. What better way to spend your New Year's holiday than curled up with a laptop learning about software?

Even Faster Web Sites with Steve Souders

In this session you learn: How to make your Web sites 25-50% faster; The impact of iframes on your Web site, including blank iframes; How inline scripts block rendering in the entire page and downloads; What you might be doing with stylesheets that make your pages twice as slow; The various techniques for dynamically loading JavaScript, and how they vary in how they affect the browser.

Advanced Web App Security with Joe Walker

The security landscape is changing dramatically from month to month. Unless you are aware of CSRF, Anti-DNS Pinning, Javascript highjacking, and the many ways to fool an XSS filter, it's likely that your Web application is not secure. Attackers used to concentrate on ActiveX, but now Javascript, CSS and even simple HTML elements have are used against Web sites.

In this session, we reveal:

* Security challenges particular to a Web 2.0 world;
* Details of CSRF, Anti-DNS Pinning, JavaScript hijacking, fooling an XSS filter and more;
* How you can protect yourself, from both the point of view of site owners and users.

The 7 Habits for Exceptional Perf with Stoyan Stefanov and Nicole Sullivan

Improvements in Web site performance are similar to improvements in energy or fuel efficiency: We make great progress, yet we end up consuming more. Learn how to balance design and features with the need for speed. This session highlights Yahoo!'s latest research results and performance breakthroughs. Apple's iPhone has changed the game for Web browsing on mobile devices. While the iPhone presents new and exciting opportunities for Web developers, it also provides a unique set of performance challenges. Solutions that reduce the number of components improve the user experience greatly by making pages load faster. In this session, we explore case studies that demonstrate how these solutions have accelerated the user experience on Yahoo!'s most prominent Web pages. In this session you learn: Performance optimizations that give you the biggest bang for your buck; Latest research results and performance breakthroughs discovered at Yahoo!; Apple iPhone's cache characteristics; How to balance features with speed.

John Resig has laid out his thoughts on a Web Developer's Responsibility, and it comes down to working with various up and coming browser versions and filing bugs:

It's safe to say that the biggest tax on a web developer is spending so much time dealing with browser bugs and incompatibilities. Thus it has become the favorite past-time of all web developers to complain about having to deal with them. Browser bugs are annoying, frustrating, and make your job incredibly difficult.

Because browser bugs are so frustrating and such a burden on top of normal development it should be the responsibility of every web developer to make sure that the browsers they develop for are able to find and fix their bugs. By taking responsibility for the bugs that you find - and to not assume that "someone else will find it" - will accelerate the rate at which browsers can improve.

The solution to helping browsers is two-fold: 1) Every time you find a browser bug, file a bug report to the respective browser. 2) Actively test your sites in the latest builds of the major browsers.

The vast majority of web developers have never filed a bug report with a browser vendor - or even used a nightly version of a browser - which is a shame. If you think about it there are few who are more qualified to assess what is going wrong in a browser than those who spend every day developing for them.

I'm especially surprised when I see professional developers not filing bugs with browsers, or testing on nightlies. Since one of the primary tasks of most developers is to paper over cross-browser issues it becomes in their best interest to see the number of bugs reduced (and making their job dramatically simpler).

Now, instead of just saying "go do it", John gives information on how to file a bug, where to go to do it, providing a good simple test case, and arguing for a bug.

He then shows examples of bugs that he has filed across browsers, and asks us all to wear them like badges.

So, got any badges? Been frustrated when filing a bug? What's the weirdest bug you have ever seen (sorry, interview question there).

The Britain from Above BBC programme and website doesn't seem to use the BBC library Glow but instead has nice Mootools effects on the front page and a jQuery Carousel on sub pages.

The effects made me feel like I was on a Flash site at first, but the real interest is the amazing visualizations that the programme has done.

Take a look at the Taxi's at rush hour:

Jeremiah Grossman, our number one Web security chap, has some interesting words as we jump into 2009:

It’s unanimous. Web application security is the #1 avenue of attack according to basically every industry data security report available (IBM, Websense, Sophos, MessageLabs, Cisco, APWG, MITRE, Symantec, Trend Micro, SecureWorks, ScanSafe, IC3). This is in addition to reports specifically focusing on custom Web application vulnerabilities (WhiteHat Security, WASC, Accunetix). SQL Injection and Cross-Site Scripting are routinely cited as the biggest issues, the ones we can’t apply patches to defend against. Perhaps what we’ve learned in 2008, as pointed out by Gunnar Peterson and Gary McGraw, is we’re spending on the wrong problem. Roughly $150MM in software security products & services versus the lopsided billions annually on network security. 2009 will give us another opportunity to make a difference.

From the mountain of statistics available I've saved several interesting quotes to reference in 2009.

He goes on to use quotes from various sources to tell the tale. It appears that most of the attacks are there to put something on your machine. With business models where the malware doesn't need to popup and try to sell you something, but rather just use you CPU as part of a botnet. The fight will continue in 2009!

Once you’re a member, you can post a photo of your daily style, your friend’s outfit, someone fashionable you see on the street, or a Creative Commons photo. Members can save, comment and vote for looks they like.  In many ways, you could consider Geomoda as a hybrid of The Sartorialist, flickr, and Digg.

Currently, Geomoda registration is by invite-only. If you have a strong interest or background in fashion, you may request and invite.