Hackweek 9 @ SUSE

It’s hackweek time again at SUSE, and we are using the time to improve SMB Traffic Analyzer. At some point we decided to move the whole SMB Traffic Analyzer software suite to QT4. This movement is fronted by Nanuk Krinner, who already implemented our realtime component smbtamonitor-qt. His work at hackweek 9 will improve the graph display, and it’s realtime movement.  I am working on a port of the object selector of webSMBTA to QT4, and a solid integration of smbtaquery with smbtamonitor-qt. Once we have reached a certain point of integration, we will work on porting webSMBTA features to smbtamonitor-qt. We found it hard to deal with framework updates for webSMBTA. Since webSMBTA is based on smbtaquery too, it won’t be dead anytime soon, but we certainly will give more attention to the QT port in future.

Check out Hackweek 9, and explore all the cool ideas people have put on the table to hack on for a week!

 

SMB Traffic Analyzer 1.2.6 released

Should you read this article in a blog roll, and have the impression what you read is from outer space, note this article is about the SMB Traffic Analyzer project, a software suite to provide statistical traffic analysis for Samba networks.

Summary: with a development time of about a year SMB Traffic Analyzer (SMBTA from now on…) version 1.2.6 has risen out of our git repositories to represent a new milestone for the project, bringing full buckets of features and bug fixes to the table. With Nanuk Krinner, another developer joined the team and brought in completely new areas of SMBTA usage, by creating a component for desktop usage. To view all the details on this release, the tracker bug for this release can be viewed here, and the list of bugs being handled by this version is: 8167 8168 8172 8205 8206 8210 8232 8237 8270 8280 8288 8298 8321 8439 8496 8497 8919 8932 8934 9061

 

The whole real-time infrastructure has been redesigned and rebuilt. We have concentrated on making the interface simpler, and splitted the infrastructure into a very simple and lightweight generic client called “smbtamonitor-gen“, that can be called by a parent process (such as a graphical client), connect to smbtad and interface with the parent by unix domain sockets on the local machine. This allows us to develop more complex real time applications that don’t need to cope with the low level networking to smbtad. For a start, we have developed two basic real-time apps, a text mode client for the command line interface, called  “smbtamonitor-cli” that does nothing else than posting traffic values on the command line and can be used as a foundation for more complex clients,

An instance of smbtamonitor-qt running against a traffic simulation. Of course it runs flawless against real servers too.

and Nanuk Krinner developed a first version of “smbtamonitor-qt“, a real-time application employing the QT library to allow a new level of usability and features on the desktop level. Although being seen as a starter, smbtamonitor-qt is already able to dynamically zoom in and out of Samba traffic in real-time, from a time range of a day, to a scale of up to 5 seconds ( bso#8932, bso#8919). Also we have fixed a crash in the smbtamonitor-gen component when a Samba object could not be identified (bso#8280).

Since SMBTA is not only about real-time, the project relies on a database to store Samba traffic that happened in the past. Here we track more detailed data, such as usernames, filenames, and ip addresses to name a few. The groundwork tool to allow for stable, complex and portable querying of this data is smbtaquery, which produces XML that allows other tools sitting on top of smbtaquery to interpret and create views out of the data. smbtaquery has relativly complex time functions that allow to limit queries over specific timeframes. The qouting mechanism in smbtaquery has been enhanced to allow more complex timestamps given as time delimiters in functions (bso#8169), and a segmentation fault when no database driver was given either in configuration or by command line, has been fixed (bso#9061).

We have fixed a segmentation fault in smbtaquery (bso#9061), and fixed a crash of smbtad when handling multiple real-time clients (bso#8934).

The new function ‘system->self-check’ shows the status of all components of the software chain, and can be run by admistrator-type users from within webSMBTA.

When updating from a former version of SMBTA, it was usually necessary to call “smbtad -C” to convert the existing database to the newer version where we have added features. Due to changes in the database scheme, that now includes integrated version control, smbtad just needs to be restarted, and it will check the database version against the format that it is running on, and convert the database fully automatically when needed. There are also new functions to smbtaquery to allow the user to check that any component is up to date. (bso#8206). smbtad will now also dynamically update the protocol version of VFS modulesthat are connecting to it. Therefore, within a “report” function, smbtaquery is able to tell the user

Time range selection has been eased by using this widget within webSMBTA.

about the status of the whole software chain, and adds comments and notes to where it applies. For example, the VFS module, as coming from a specific Samba version, might already implement a few new features that are not yet used in SMBTA by smbtad and the rest of the tools. These features might be added later to the rest of the software suite, and smbtad simply ignores these features to stay compatible. The report function in smbtaquery is able to inform the user about this status. Last but not least, SMBTA 1.2.6 can also self check online for a new version of the software package, so that the user will be informed about the availability of a new version of SMBTA on it’s website.

Administrators will have a new function that allows to retrieve the configuration settings of a remote running smbtad instance. There are two versions of it, one returns a fully commented list of configuration settings, another is in a simpler format just listing all the configuration values. This function is of help in bug reports. It can be called from within webSMBTA.

Upon user requests, we adapted smbtad to also run on sqlite3, again. This time via libDBI, and all our initial tests worked fine as far as smbtad is involved. Tests using smbtatools are still pending and any user who wants to join testing smbtatools for sqlite3 is highly welcome (bso#8288).

The webSMBTA component now integrates user management and authentification, allowing for

The administrative interface in webSMBTA allows to add/set/remove admin users and normal users.

users that have administrator-rights and normal users. Functions such as the remote configuration report of smbtad are only able to be used by administrator-class users. webSMBTA allows full user management within the web interface. More than one admistrator-class user can be added, and also as much users as wanted. (bso#8167).

In addition to this, webSMBTA stores the functions that a user has been running, and is able to restore the “workplace” of every user when logging in (bso#8496). Every function callable in webSMBTA now has a way to produce alternative output. That means, not only HTML, the user can also request the output in ASCII (or any other format we might support in future), and webSMBTA will offer the alternative format as a download. Related to this new feature, it also offers a print-view on every function it is running. (bso#8172). The webSMBTA functionality to select time modifiers has been enhanced and allows comfortably to select time ranges by using a calendar widget.

webSMBTA now has a small input field that triggers a search function as you type…

One of the really cool additions to webSMBTA is an integrated “global search”.

… the search function will immediately be displayed and updated as you type, runs over anything SMBTA has seen, identifies what objects it has found, and makes those clickable….

The user can just enter free form text and webSMBTA will search through the whole database while typing, and identify each objects it is going to find. Clicking on one of the found objects will automatically set the object chooser of webSMBTA to point at this object. This function greatly eases usability and offers a fascinating

… and a click on an object triggers webSMBTA to mark the chosen object in it’s selector for the user to further examine the object!

way to search through a Samba network (bso#8237, bso#8298). To easy the way webSMBTA is handling the database, a “–test-db” option has been added to smbtaquery, allowing to only test the database connection, and do nothing else. This allows for better error handling within webSMBTA (bso#8270). For it’s internal operations, webSMTA is requesting some space to temporary store some helper files. The path on the filesystem to use these helper files has been hardcoded to /tmp in former versions, now a variable temporary storage path can be given by setting the Dir.tmpdir variable inside of webSMBTA (bso#8497), wich is of help on systems where the usual /tmp path is not reachable or an other temporary storage path has to be used. On a side note, webSMBTA has been ported and tested on rails 3.2 during the Hackweek event at SUSE in 2012.

Our project documentationis no longer available within the package.

The whole documentation set for SMBTA has been moved to SambaWiki.

Instead, Benjamin Brunner converted the whole documentation from asciiDOC to wiki format and moved it to SambWiki in an effort around the Hackweek8 event at SUSE. This will allow developers to quickly change the documentation in future, a motivational plus when it comes to maintain the documentation in relation to bugs and features Many thanks to Benjamin for this large effort, it marks a starting point for us to move SMBTA to a new homepage, which is in our plans.

We have once again changed the database format, this time for good hopefully. We have simplified the database format a lot to help making new features much more trivial and easier to implement. Additionally, we store more dynamic data in the database while smbtad is running, such as the number of connected modules, and from where (from which IP) they come from (bso#8205). A critical bug in smbtad has been fixed, that now assumes that any filename or path entries in the database have to be qouted. In former versions, filenames such as ‘   “hello world”  ‘ could have caused problems when querying the database (bso#8232). smbtad now allows to bind to an interface address and works as well as within IPv4 and IPv6 networks (bso#8321).

Benchmarking and performance

Performance tests with a file size of 1MB

tests have been run to show the speed penalty one gets from using SMBTA. By using the Phoronix testsuite, Robert Piasek has run a few tests employing exactly what smbtad was optimized for, and we have added the results to the package. Performance loss is one the most asked questions when we are doing presentations around SMBTA everywhere (bso#8210).

SMB Traffic Analyzer is free software, released under GPLv3. For more information, please visit it’s homepage.

Teradata Partners Conference / Novell Brainshare / SMB Traffic Analyzer / Development status

Joshua Tree National Park. Full of wonders.

"Mutter und Kind" @ San Diego Zoo

Well last weeks have been pretty busy. With a 10 hours flight, we arrived at San Francisco to get to San Diego to visit the Teradata Partners conference, an excellent event that included good discussions and lots of acceptance and recognition for SUSE Linux. I liked the way talks have been organized, or to say it in other words it was a matter of downloading an Android app to always have the overview. We served the SUSE booth at the conference, got involved into lots of discussions about the OS, and I was even happier in the moments when people asked questions on openSUSE – this is showing clearly that the project is well known! To leave some words on San Diego is very simple: Gaslamp Quarter, I’ll be back, and have a few more beers at Rock Bottom!

We then drove from San Diego to Salt Lake City by car. Having some days of vacation in between, this was the best choice. During our travel we stopped by and visited San Diego Zoo, the Joshua Tree national park, Zion national park, and Bryce Canyon.

In Salt Lake, Novell’s Brainshare conference was about to start. And guess what I’ve had the chance to present SMB Traffic Analyzer at this amazing conference! I’ve had the presentation two times actually, presenting the very latest development snapshot of SMBTA and while it was not visited by so many people, the ones who listened have been very tough and showed great interest, giving me the chance to make new contacts, and lots of discussion about Samba and SMB Traffic Analyzer, and openSUSE. So to bring this to an end: Brainshare was exciting!

SMB Traffic Analyzer is developing healthy. I was presenting Benjamin Brunner’s work on the real-time search function, user management, and the very latest statistic functions in webSMBTA. We haven’t had a release in the last months, which is unusual for SMBTA. That is actually caused by features we want to implement for the release. These features, such as SVG graphics for usage diagrams are not yet done,and I am working on it.

SMB Traffic Analyzer @ openSUSE conference

Should you read this article in a blog roll, here is a clue what it is about: a) the SMB Traffic Analyzer project (SMBTA), b) the openSUSE conference, and c) Samba.

You guess it, we finally got a slot for a presentation at the wonderful openSUSE conference for SMBTA. To me, it is remarkable to see a project like SMBTA being presented at OSC because it is not really something related to openSUSE. It’s not that SMBTA improves your boot time, or discusses details of the buildservice, or makes your life with the openSUSE distribution better in any way. SMBTA is very likely not even interesting to the casual user, except for some administrators.

That said, SMBTA was born inside of the openSUSE infrastructure, growing to a project used on different distributions and operating systems, such as Solaris. And the one thing we can really say is that we exploited all the services that make up openSUSE to the core. We used the openSUSE Buildservice from the beginning, and we use appliances created by SUSE Studio for both demoing and developing SMBTA.

With the recent release of Samba 3.6.0, among it’s top changes like full SMB2 support and other major features, it is also prime time for SMBTA. The Virtual File System layer module that supports our current infrastructure is included in this release of the Samba CIFS server and that marks a milestone for our project. SMBTA is already used in production at some sites, and the release of Samba 3.6.0 will hopefully forward this trend.

Benjamin Brunner and me will give an introduction talk to SMB Traffic Analyzer at the openSUSE conference and most likely live-demo the software chain. We’ll welcome anyone interested to join our presentation at OSC!

SMB Traffic Analyzer submitted to openSUSE Tumbleweed rolling update distro

The complete SMB Traffic Analyzer (SMBTA in the following) software package (up to date Samba including SMBTA VFS module, smbtad and smbtatools) are now available in openSUSE’s Tumbleweed distribution, making it ready to run SMBTA out of the box. Since my blog is read by quite some people outside of the openSUSE community, I am using this space to shortly introduce the Tumbleweed project, as it might raise interest to those who were not aware of it.

openSUSE Tumbleweed is a project to provide the openSUSE Distribution in form of rolling updates. At a glance, this means in the ideal world: Install Tumbleweed once, and never run out of updates or maintenance, as Tumbleweed will update itself to always match the current package versions that openSUSE’s maintainers or the according open source projects consider as the current stable release of a package. In difference to the openSUSE Factory distribution, which simply provides the current development state of openSUSE, Tumbleweed will not provide unstable or experimental packages. The projects idea was being discussed for longer time in the community, and finally brought to life and implementation by Greg Kroah-Hartman, who announced a Tumbleweed repository in the openSUSE Buildservice. In it’s current form, Tumbleweed is driven by a number of package maintainers and developers who want to make Tumbleweed to happen, and relies on package maintainers to support Tumbleweed by submitting their packages to the repository.

It’s very easy to jump from openSUSE 11.3 to Tumbleweed by using zypper, to give it a try. I run it successfully in a virtual machine since Tumbleweed was announced, and consider to make it my distribution for development/work as it brings advantages that I don’t want to miss. Being able to release a new SMB Traffic Analyzer version and having it available for users and packaged in a distribution in a matter of hours is one of them. Read more about that on the Tumbleweed page in the openSUSE wiki.

SMB Traffic Analyzer 1.2.2 released

After some delay which wasn’t on our plan, we’re happy to announce SMB Traffic Analyzer (called SMBTA in the following) version 1.2.2, bringing a whole bunch of new features and bugfixes to the table.

First off, you might not believe it but SMBTA actually received some artwork, as we already reported here.  Version 1.2.2 includes those graphics as well as the SVG source. Thank you Sirko!

SMBTA was installed on completely different systems than Linux during the past months. In some cases, things could have been much easier if required libraries could have come included with SMBTA. So we did for sqlite. If shared sqlite objects are found, and are of version 3.7.0 or newer, those will be used, otherwise SMBTA will compile the 3.7.4 Amalgamation build of sqlite, that is included with both smbtad and smbtatools (bnc#662186). By specifying an environment variable, the user can also force an amalgamation build, even if shared libraries are available. Since the amalgamation build of sqlite is basically sqlite in a single c file, most compilers can employ optimization much better compared to a usual sqlite build.

We are in the process of addressing possibilities to make SMBTA more than just output it’s results to a text mode terminal.

A run of 'global, top 10 files rw', and it's HTML output

The decision was taken to drive the output of smbtaquery – the main program used to query the database of SMBTA – with XML, a move that allows us maximum flexibility. The smbtaquery program therefore produces XML by default and employs the XSLT processor xsltproc of the libxslt library to convert the output to HTML or ASCII text automatically.

Snipped of the "usage" function output in HTML

All functions of smbtaquery are supported in both formats. SMBTA includes style sheets to produce the output and the user can choose the prefered output format on the command line. HTML support in smbtaquery forms the core of our plans to create a web based user interface for SMBTA (bnc#659326).

SMBTA 1.2.2 extends 128Bit AES encryption for the whole software suite. All tools, smbtad, and the module are now able to talk encrypted to each other. With former versions this has only been possible from the VFS module to smbtad. To support the end user, smbtaquery has been extended to generate 128 bit keys for usage with smbtad. The smbtad daemon supports two different keys, to allow a different group of users running the smbtatools (bnc#599644). A long standing issue finally resolved.

A lot of effort moved into our test suite – smbtatorture -, in order to enhance the SMBTA Stresstest appliance. First off, a smbtatorture process now creates directories, and filenames that make some more sense then the ones used before (which were just generated out of the user name and a number), to produce much more realistic looking results (bnc#653618). Furthermore, we worked on having support for multiple instances of smbtatorture on the same shares. A small control server has been created, “smbtatorturesrv“, distributing unique filenames to make sure that no smbtatorture process chooses the same file name than one of the others running on the network. smbtatorturesrv is internet socket networked, therefore smbtatorture processes can connect from anywhere to it.

Snipped of the SMBTA-Guide. We will work on getting more illustrations into where it makes sense.

This program will be extended to make up a controlling instance for the connecting smbtatorture processes, showing statistics and healthiness information. We will implement the features introduced with 1.2.2 in the upcoming version of the SMBTA Stresstest.

On the documentation, probably the most irrelevant part of SMBTA , it has been completely reviewed and many design changes have been made. Everything about encryption has been taken out, and formed into a separate new chapter, addressing how encryption works in all parts of the software suite. All new features of smbtatorture and smbtatorturesrv have been documented. We have added descriptive illustration and diagrams to the documentation. (bnc#664823). And of course we updated the online version of this document.

Oh wait, the openSUSE’s BuildService already consumed SMB Traffic Analyzer 1.2.2. We will submit packages to openSUSE Factory and Tumbleweed in the next days.

You can download SMB Traffic Analyzer on the Download Page. To get more info about SMB Traffic Analyzer, visit it’s home page.

The openSUSE board elections are proceeding

It’s time to vote a new openSUSE Board, a group of six people helping the project overall by resolving conflicts, standing as a central contact point, and actually making decisions where it’s required.

Just gave my two votes for the candidates I prefer, to make sure they get at least one vote. Very detailed information on all candidates can be looked up at this page.

All those with openSUSE membership status are allowed and called to vote. Please check out the candidates and do so !