InterNetwork News / INNX

In 1999 my brother Dan Thorman and I amicably exited from our successful partnership with Ignition Design and joined the former management team of Parador Productions to create a new angel-funded startup company to produce TV health news for simulcast on the Web and local news stations nationally.

Over 450 top-quality health news stories were produced, airing nearly every weekday for 2 years. Each story was seen by approximately 9,000,000 NBC viewers around the country, as confirmed by monitoring reports.

INNX news items were followed by sponsored on-air “billboards” that led viewers to the Web to follow up on the story. On a Web page branded with the TV station’s identity and featuring links to the advertiser, visitors were able to view the video again in any of 9 streaming video formats, read a publication-quality 1,000-word illustrated print article on the topic, and follow dozens of links to other Web resources.

The company’s HealthSurfing.com and co-branded sites rapidly became among the most-visited health news Web sites in the United States. HealthSurfing.com was regularly ranked in the top 10 health-related sites in the country by several independent sources. The video content was even syndicated by Yahoo and several stories and still images were picked up by major national newspapers.

The parent company INNX (later renamed Traffiq Systems), through its editorially-independent news division known as InterNetwork News, produced all the news segments and also all the Web content. (For convenience, I will refer to both entities collectively as “INNX.”)

As co-founder, CTO and lead engineer, I supervised, designed, and personally implemented every technical project in both divisions of the company, from the multiple Web sites and customer-support applications, to literally dozens of internal process-automation applications in support of our demanding production and publication schedule and business operation needs.

Our business model was to sell advertising time adjacent to the TV news, and since we could predict the content of the our own a few days in advance, we could achieve a premium advertising price as compared to non-targeted advertising rates. By sharing the premium with the local TV stations and increasing their ad sales while simultaneously reducing their own TV news production costs, everybody won.

Unfortunately, the company ultimately was forced to close due to lack of ability to raise funds in the aftermath of September 11th, and declared bankruptcy in November 2001. However, the TV advertising-adjacency ideas developed by the company became the basis for 2 new startup companies in which I opted not to participate.

Every system at INNX was built using 100% open-source tools, including: Linux, apache, perl, mod_perl, Embperl, mysql, ssh, rsync, and dozens of other open-source tools and perl modules written by others and by myself. The majority of desktop computer users at INNX used Macintosh computers, and a few used Windows. All file services were supplied using Linux-based file servers and open-source samba and netatalk software, resulting in complete cross-platform access to all corporate data from Mac, Windows, and Unix applications. All backend systems were implemented on Linux. The result was a highly-integrated heterogeneous computing environment that enabled a phenomenal degree of productivity by a comparatively small staff.

The items below outline just a few of the projects I designed and developed at INNX in the period from 1999-2001.

News Content Management System

Every day, INNX needed to produce a 90-second news story for TV broadcast and a corresponding 1,000-word print story for Web publication. While each TV story was the product of at least 6 people ranging from producers, on-air talent, to video editors, the Web stories had to be created by one writer with occasional freelance assistance, edited and proofread, and produced for the Web by the company’s web production designer who also had dozens of other design responsibilities.

Since content was released daily, an complex production pipeline was needed to track each day’s story or stories from concept to writing to editorial approval to the automated nightly Web-push. To complicate matters further, the team members were not located in the same cities.

Solution: I designed and implemented dozens of content management procedures for the San Diego-based TV production office which allowed team members there, all working on Macintosh, to upload all video and other content to a Linux-based file server.

Once uploaded, a variety of automated processes replicated the video and other content first to the main data center in Redwood City, CA, and from there to the colocated Web server farm in San Francisco, CA. Automated e-mail-based reports were generated allowing TV production teams in San Diego and around the country to preview and approve the video over the Web in rough and final cuts to ensure highest standards of editorial accuracy and review.

Process automation tools were written in perl and ran primarily on a large network of Linux machines linked via a multi-city masqueraded tunneling VPN (using homebuilt Linux-based firewalls) with key-exchanged ssh for RPC. This way, a single master server was able to control and automate processes on servers at 4 different facilities, all with unbreakable network security.

Finally, a number of Web-based Intranet applications were created to allow supervision of the Web sites and production processes.

Of these, the most important was a story-content management system that allowed the Web writer to work on news stories in a semi-structured text-based format, suitable for exchange with other team members via e-mail. The writer then checked a story into the system by uploading it to a Web-based application which then performed hundreds of cleanup and formatting tasks, checked for errors, filled in missing and derived fields, and then generated e-mailed notices to the producers responsible for the stories so they could sign off on the content and the editors and web team so they could prepare the requested images. The system allowed for multiple revision histories of each story so changes made at each stage of the editorial process could be tracked. The system also allowed real-time previewing of the fully-rendered Web version of the story and its linked images and video assets by the writer. This was possible because all HTML markup was generated directly and in real-time from a parsed and cached version of these story templates. No HTML markup for Web stories was ever created by the writer or the Web staff.

The network design, process design, server configurations, and all automated tools and Web applications were created and iteratively refined by myself personally and received daily use by every person in the company.

INNX database system and schema

The INNX corporate data model (SQL schema) needed to include dozens of objects - including: stories, pages, clients, stations, advertisers, ad rates and terms, Nielsen audience data, INNX broadcast schedules, local TV broadcast schedules, people, and much more. It needed to support dozens of Intranet and public applications, from story production, to Web search, to business procedures, such as tracking hundreds of TV stations and dozens of clients.

The database needed to be transparently distributed, for security, redundancy, and failover purposes, across several database servers, some located at secure firewall-protected sites and some at an offsite Web colocation farm.

Solution: A distributed database system was implemented in mysql with schema and a sophisticated administration tool created by myself. The administration tool, written in perl, formed the foundation for all automated data-replication processes as well as safe, replicated, changes to schema (which took place frequently and did not require any downtime).

All access to the database objects and the corresponding business logic were encapsulated in object-oriented perl modules, which were in turn used by all applications throughout the entire INNX system: process automation tools, and Web apps on the Intranet and public Web servers.

A master database server located behind the corporate firewall held master copies of all data tables. The system tracked the “master host” server for each table, and was thus able to appropriately server, replicate, and/or back up each of the following classes of tables:

• Private dynamic tables hosted internally — modified by web-based or other intranet applications; backed up internally and not replicated to any insecure colocated servers. Examples: Client data, invoices.
• Public dynamic tables hosted internally — modified by web-based or other intranet applications on the master server; backed up internally and replicated to colocated servers for access by Web applications. Examples: TV story templates, headlines.
• Public dynamic tables hosted externally — modified on a centrally-designated external master host by public web-based applications; backed up via regular replication to the master server and also redundantly replicated to other colocated servers to allow for failover so any other colocated external server could take over the hosting of dynamic master data at any time. Examples: TV Airing Schedules entered by local news stations, TV station preferences, cookies.
• Tables created and loaded from static files — loaded as needed into mysql on the firewall-protected master server and replicated to all slave servers as appropriate, allowing for local mysql access by each Web host. Examples: TV stations, time zones, Nielsen viewership data.

Several clever auto-homing subclasses of my open-source module DBIx::TableHash were created so all table objects used by any INNX application knew which database server to connect to in order to get to the most-local copy of static tables, or the master copy of any dynamic tables, allowing zero changes to application code when database-hosting changes were made, again allowing for easy failover.

Interactive Web applications

While all TV stations were well-equipped to receive and broadcast INNX televised video stories, they were all ill-equipped to deliver the corresponding Web links from their local TV station Web sites to the INNX-produced Web pages for each news story. However, being able to deliver this enriched Web content via their own Web site was a primary source of brand-building value for the stations.

Solution: INNX created the sophisticated INNbox and the Story Management System (SMS) — an integrated suite of Web applications that allowed TV station producers and webmasters to easily set up access to a co-branded Web site where INNX-produced stories could be viewed by viewers.

Though the co-branded story sites and video were served by INNX, the look and feel of the pages could be controlled by the TV station’s webmaster by logging into the SMS and following a simple set of instructions. When ready, the Webmaster simply copied and pasted a block of HTML code into the station’ own home page to create the INNX-delivered “INNbox” — a dynamically-generated graphical control panel listing all the headlines of INNX stories recently shown on that station, with links to the co-branded pages.

Once the INNbox and co-branded design was in place, TV station producers simply logged into the SMS, clicked to indicate which stories they planned to air (and when), and the rest was taken care of by INNX — the INNbox auto-updated at precisely the correct local time in the TV station’s market to show the Web links viewers were directed to on-air. Viewers who clicked the links were taken seamlessly to the INNX-served co-branded pages, and when done browsing, simply clicked any of the station-provided navigation elements or back links to return to the TV station.