ByteSized VPS
As soon of you might have picked up we've been working hard at our new sister-service ByteSized VPS. As we will soon be starting a more open Beta for existing ByteSized members we feel it's time to tell you a bit more about what a VPS is and who might be interested in one.
Let's start with Wikipedia's definition of a VPS:
"A virtual private server (VPS, also referred to as Virtual Dedicated Server or VDS) is a method of splitting a server. Each virtual server can run its own full-fledged operating system, and each server can be independently rebooted."
A VPS is a server in a server as it were, it operates as a dedicated server; you have complete administrator access, can reboot it, can re-install it and can choose what ever operating system you would like, it even comes with your own dedicated IP.
Now why would you want a VPS?
- Complete freedom to do whatever you would like, since you have administrative access there is no limit to the applications you could run on it. Want to host a website; install a webserver. Want to setup an irc-server? Then do so!
- Better resource management, the technologies that make VPS's work comes with a whole set of resource management tools this way we can make sure each user gets the resources it deserves;
- Dedicated IP; some trackers still have problems from connections from a shared ip, having a dedicated IP resolves this issue.
- Better monitor tools; You can check out how much bandwidth you used this month, what your load averages are like and how much memory you are using.
- Ready to Rock Templates; Because we know not everybody is a Linux guru but would like to use a VPS because of the other pro's we created two ready to rock Desktop/Seedbox templates. Setup with rTorrent/ruTorrent / Deluge or Transmission and all kinds of other handy tools. You can check out the wiki for screenshots and application list.
Now sadly it's not all fun and games and there are some cons as well.
- It's more expensive; IP's and the new management system means higher overhead costs.
- Less support; With great power comes great responsibility. The stuff we deliver in our custom templates should work out of the box and if not we will provide support for it but everything that goes beyond this, you do at your own risk. We'll do our best, but we can't provide support for every application there is.
Once we go out of BETA we should have VPS's in different countries, so far we are trying to get servers in France, Luxembourg, The Netherlands and The USA.
Existing members should keep a close eye on the news. We will post initial pricing soon and we will also start adding some more people to the BETA.
OpenVPN + Open Registration == WIN?
We've been very busy with ByteSized VPS (more on that in a later article) over the last couple of months and because of this we didn't give our ByteSized Boxes the love it deserved. That's why today we have a little freebie for all our current and future users.
All our boxes will come with VPN-access from today on out. VPN is a way for you to conduct your internet business in total anonymity by routing all traffic on a secure connection to our servers. To use our VPN-service you will need to install and configure an OpenVPN client. We have some basic instructions on this wiki-page. Something else we decided to install for everybody is SABnzbd.
Another announce we have for you today is that from this moment on we will be having open signups for a while. We might make it permanent or just a few weeks, we don't know yet. But for now tell your friends they don't have to wait around twitter any more. Because of this we can't guarantee our "within minutes" setup time anymore, instead we will announce it on twitter if the setup is going to be longer then average.
We are also working on a new wiki-system, we are leaving the custom-made wiki for a more open wikimedia-wiki. This new wiki will hold both the articles for bytesized-vps and bytesized-hosting. Moving all entries from the old to the new wiki might take some time and will not be done in a day, so for now just use the old wiki till we moved all stuff accross. We do want to ask you that if you want to create a new page do it on the new wiki which can be found here.
More information on Bytesized-VPS will come in the following days, for now you can find a short F.A.Q. on the Wiki.
Overall price increase
Today we have some sad news: We are forced to raise all our prices by 1.5 euro per plan as of now.
Let us explain how this came to be:
A few weeks ago we've started noticing some of our servers getting less network throughput than expected. At that time, we investigated all possible hardware and software issues we could come up with, but couldn't find anything that could cause this kind of behavior. After multiple inquiries to our provider regarding the issue, they finally told us that our servers were using too much bandwidth and because of this, they were being throttled! Although their plans and terms of services say that they offer unlimited bandwidth, this isn't actually the case. Since, contrary to common belief, bandwidth is not free we understand that providers offering unlimited bandwidth have to come up with a mixed calculation. They expect that some users use alot of bandwidth while others compensate for this by using less, since someone has to pay for all that bandwidth in the end! The problem is, the calculation isn't working out for them currently. Not willing to compromise the quality of our service, we have now negotiated a new deal with our provider to guarantee completly unthrottled servers. This comes with a price though.
We do understand that a lot you decided to go for us because we provided a really cheap service and we very much regret that we can't afford to sustain our current pricing scheme anymore. We do hope however to have proven that we are not only cheap, but also strive to offer the best service out there. If anything, this situation has made us even more eager to look into other solutions, we could bring to you in order to enhance your ByteSized experience.
One of the first new additions to our range of products will be the launch of our VPS service, which will feature less people per server, better resource management and more customization options. Tailored for those users who want a little more bang for their buck. We expect the public beta to start in the following weeks.
We hope that you understand that where we come from and we feel that this was the only option to sustain our level of service.
Upcoming website changes
We are in the final stages of deploying the new website code but before doing so we wanted to make sure everybody would be comfortable with the changes.
The main thing that changed is that one (website-)account can now have multiple seedboxes under his control. This has a couple of pros.
- When you cancel your box, you can still keep your website-account and the ability to request new boxes when you need one again. In a way rewarding ex-members with easy access if they ever want to use our service again.
- We can keep better track of box and member migration
- It gives us the ability to easily add new services..
To make this all work the site will experience a small downtime either tonight or tomorrow to run the migration scripts. All your boxes should still be accessible during this time.
Although the changes are very apparent here is a small screencast to help you along.
If you want to merge your current accounts under one account come find us on IRC, after the update is live. If you have a specific question about your account please create a ticket and don't use the comment section here.
ByteSized Setup – The Website and Daemon
Previously we gave you an overview of our setup, this time we will go over all the website functionality for both the user and admin users.
We wanted to have a unique feeling without using a generic template and spending money on a real designer ;)
This is one of the early sketches we made, you can see some parallels with the current design although not much survived. As you can see we were pretty nameless back then.
After this inspiration-shot the actual building started it took two weeks to get the first version live that had user registration and a simple payment system.
After this small history lesson let's continue to the main feature as of this date, this part might not be so interesting for current members as they should have access to everything explained here, feel free to skip it. (Allmost) all data you will see in the screenshots, which can be clicked to enlarge, is fake, so don't bother trying to login with it ;)
One thing that might be new to current mebers is the second menu-bar, but more on that later. What you are seeing here is the main-screen of the website; The Dashboard. All the important information is on there, from box information till your latest tickets. If everything goes well this is the only screen you would really need. The first connection with the bytesized-daemon is made here, when the dashboard loads it will try to get your quota information from the server you are placed on, hekatonkheires in this example. Ff it somehow fails it will fallback on the latest quota it got from cache. The same goes for the service status, only there is no cache mechanize for that yet, so if that fails you will just see a message explaining that the server did not respond in time. The reboot box option runs your startup and stop scripts. These reside in your home folder and tell the server what to do on reboot. By default it just kills are your processes gracefully an starts them again, but you could do all kind of stuff with it if you are a little apt with linux.
The "Balance" item takes you to a page where you can see all kind of financial information, how much money is on your bytesized-account, when you need to add money again, and when your next invoice is due. Next in line is the "Tickets" page. This is the place where people can go to for their questions. We make the difference between private and public tickets. Public tickets are viewable by everyone, this enables members to help each other out, or discuss certain topics. Private tickets are only viewable by admins and are primarily used for payment issues.
The "Wiki" page is just that, a wiki filled with information on general website usage, building a custom version of your favorite client or on setting up a FTP client. Everybody can add new/edit pages and so far it only got wiped once! At the "Account" page you are able to change your password, email and other account related details. "News" is an archive for news that isn't visible on your dashboard anymore, since that only displays the last three items. "Blog" is a simple link to, well you guessed it, our Blog.
Now the admins got a dashboard as well, and this is what's on there. It's not easy on the eyes, but as only two users have access to it, it doesn't matter much, for now...
Free spots gives us the intel on which harddisk have potential space left. You can see that Apate has 160gb left, which is enough for one Lunchbox.
After the free spots is a list of users who are over due. Usually we give them one "free" day to come up with the money, but if they don't we can delete them with a press of a button. This is where the daemon kicks in again. When we press delete we mark the user as "beign deleted",that's why there is a option for those on the dashboard as well,a request gets send to the server the user is one and the daemon calls the slicer and tells it to delete the user. Since it does a global search on the server for the users data this can take a while. When it's done it sends a message back to the website that the user is deleted from the server, the only thing left to do for the website at that point is delete the user from the database.
Since we allow users to prepay it's important to keep track of that money, folding open the "Top pre-paid" will show us who has what kind of money on his account. Slice breakdown shows us which plans are most popular, the lunchbox plans win, and how much turnover each plan has.
Users being deleted are users that are flagged for deletion but that the server did not delete yet. This is a safety measure so we don't "forget" about users if the deletion tool fails, which it luckily hasn't done yet.
If a new user signups up for our service a new box pops up on our dashboard. 
Once that user turns to "paid" we are ready for deployment. 
We are able to choose a server and harddisk for deployment. Once we found a suitable server, and selected the right harddisk it's deploy time. The website contacts the daemon on the Erato (in this example) and relays all the information to the slicer. Which client should be the default one for this user, how much space is allocated to this plan and which harddisk should this user be created on. Once it's done, just like the deletion, it posts back all the data it created for the user so it can be stored in the database.
Next up is the user database of the website. This gives us quick access to all the data we need to help out with support. Viewing a user gives us the dashboard information of that user plus extra information, all the tickets this user created, and admin-visible notes. It also gives us the ability to redeploy users, if they feel they messed up their configs, or want to start out fresh we can redeploy
them, deleting everything and rolling out a new box.
News is pretty much just that, the ability for us to create news item. Nothing to tell about that one.
Now the server overview gives us some handy information. How much usage it today and yesterday, what the average turnover/profit per server is and how much theoretical space is left. To calculate those things we need to specify a few things when adding a new server to the website. How much harddisks it has and how large they are and how much a server costs on a monthly basis.
When viewing a server we get more specifics. What the current server-load is like, how much space is left, who are deployed on said box and on which plans they are. This info is especially handy when we are looking for suitable servers to deploy users on. It gives you everything you need in an overview.
The "Finance" page gives us some really rough numbers, like how much turnover we have on a monthly basis, how much profit we have per server or user we make on average. I say really rough because it doesn't account for specials or time a box is empty etc. It's just a momentary snapshot.
"Plans" is where all our plan information lives, although there are a only three plans active by default we have loads more. A plan is fairly simple. It just needs a few details; How much space does this plan have, how much does it cost and should we display it on the frontpage/sign-up page. The space is important when deploying a user because this defines the quota set on the servers and price is used when creating invoices.
That's all for today, I hope you enjoyed reading and if you have any questions/suggestions please leave them in the comments.
Why uTorrent on Linux sucks
µTorrent is probably by far the most popular bittorrent client around, it's what most people on windows systems use at home, so it's what alot of people want to use when they order a seedbox, simply because they're used to it and know how to work it. This is understandable of course, who wants to go to the trouble of getting to know a different program when you can use one you already know? Now, every now and then someone comes into our irc channel and is suprised when everyone flinches when they mention they're running µTorrent through wine on their slice. This article will hopefully provide a link we can send to these people to make it a bit more clear. So, while running µTorrent in this way generally works, one of the main drawbacks of doing this is that since it's a closed source windows program, you need to run it through wine to get it to work on linux and along with that comes a wold of pain:
- Resources: Our stats, collected over roughly 400 users on around 50 servers show that µTorrent uses more cpu aswell as more memory then any other client we offer (as of now the alternatives are rtorrent, deluge and transmission) . It's worse in both categories than the next best one, deluge, by a whopping factor of ten.
- No proper logging: Another real problem with it is, since it's closed source and lacks even basic logging abilities on the command line, it's a real debugging nightmare, as I have personally experienced, but more on that in a different article :)
- Autoupdate can break it: Since we've got the binaries in a single system-wide accesible directory, as you usually have on linux, autoupdating (trying to replace the global file) goes horribly wrong, spawning a new process called utmpsomething.exe which consumes 100% cpu, doesn't do anything and blocks µTorrent from beeing rebooted until it's killed. With a default of asking the user on any boot if they want to autoupdate, this has cost countless hours of support.
- Port problems: In wine, there's an issue when you close an application that was listening on a port (as for example the µTorrent webui does). Wine blocks the port from beeing used again for a few minutes. This results in a reboot of µTorrent unpredictably sometimes leading to an unreachable webui, even worse, sometimes this even randomly happens while µTorrent is running! And the solution is insanly annoying: Open µTorrent, change the webui port up by one, change it back, and you're done. Since µTorrent is closed source, there's no way to look behind the scenes and to get it to work. Also, there's no way to write a script or something that can fix it properly, since it's gui-only.
- Ugly Config File: µTorrent uses a config file "settings.dat" which is serialised by bencodeing it. While bencoding is in theory somewhat human-readable, when attempting to edit the file with anything but specificly crafted editors, if you make a small mistake, hell breaks lose and µTorrent discards the whole config file. A space at the end of the file is enough to break it. Compare this to deluge, rtorrent, ...
- No wine "support": Since running µTorrent on wine is in no way "oficially" supported, it can happen that an upgrade breaks µTorrent, as this is at the time of writing the case for the newest µTorrent version, 2.0.
- No way to gracefully exit: Usually when you kill a process, there's more then one way of doing it: On linux you've got SIGTERM which you can send to process to allow it to exit gracefully, saving it's data and doing shutdown related things. If that doesn't work and program stops reacting you can switch to SIGKILL and force it to exit, potentially losing important state information. µTorrent doesn't care what signal you send to it, it just dies without saving settings, freeing ports or doing anything you'd expect it to do.
Note: I don't want to say µTorrent itself is a bad or inefficient torrent client, also wine is a really epic project and getting windows apps like µTorrent to work on linux is an amazing feat, what I want to say is, that running µTorrent through wine is a really bad idea when you draw the comparison to other clients. It works, yes, but using it is really not a great idea.
Most of the above issues can be summerized pretty well as: It's simply not a unixy program. No real commandline options (there are a few, but nothing really useful is in there), no proper config files you can simply edit in a texteditor without breaking anything, the port problems with wine, builtin autoupdate (not needed, since you've got a package manager on *nix!), etc. It just feels completly out of place when running on a seedbox, especially when you come from a mainly unixy background.
The Beauty of FlexGet
On of our latest discoveries has been FlexGet. In the author(s) own words; "FlexGet is a multipurpose automation tool for content like torrents, nzbs, podcasts, comics, etc". What this basically means is that FlexGet work it's magic on most sources and do all kind of funky stuff with it. It's RSS (among others) on Steroids. In this intro I will show you how you can take a feed and filter it with one of the most fun plugins; the imdb-plugin. I will not cover the installation bit because that will differ a lot between OS's. They have an excellent install wizard on the wiki just follow that and you will be fine.
Now to get started open up your config.yml file. The YML standard is ident-sensitive so make sure you don't mess up the indentation, use two spaces for each level. Now for this little how-to we want to parse an rss feed with movies, check it against imdb and if it matches our preferences we want to download the .torrent-file to a watch-folder.
Let's start by defining an new feed:
feeds:
my_movies:
rss: http://rss.my-movies.net
FlexGet divides plugins into three separate categories; inputs, outputs, filters. We created a new feed with the name "my_movies" and activated one plugin called "rss" which is an input filter that can parse rss-feeds. Now let's continue and add the most important plugin, the "IMDB"-plugin.
feeds:
my_movies:
rss: http://rss.my-movies.net
imdb:
min_score: 7.0
min_votes: 5000
min_year: 1980
reject_genres:
- horror
- documentary
- musical
- biography
Now I guess you can guess what most things do just by looking at the config. "min_score" defines the minimal IMDB score the movie must have to go to the next step ("min_votes" in this case). If one of those checks fails it will just stop and wait for new entries to parse. There is a lot more you could possibly do with this plugin, filtering based on language, actors that play in the movie or even who directed it. The wiki page on the plugin will tell you all about the usage.
Now to turn this into a working config we just need to define a output filter.
feeds:
my_movies:
rss: http://rss.my-movies.net
imdb:
min_score: 7.0
min_votes: 5000
min_year: 1980
reject_genres:
- horror
- documentary
- musical
- biography
download: ~/torrents/watch
The line we just added simply defines that if a match is found the corresponding .torrent-file should be downloaded to the "torrents/watch" folder in my home folder. It's a folder that my client just watches for changes every torrent in this folder will be automatically downloaded.
And that's it, a working, simple, FlexGet config. Please visit the FlexGet site and see how you can make it work for you.
Update
For those on Deluge, and hopefully that is most of you who are reading this, NimbleRabbit had a great suggestion; using the Deluge plugin. This will replace the watch folder output plugin and will add the torrents straight through Deluge. This is a great way to stay organized if you are using multiple feed.
feeds:
my_movies:
rss: http://rss.my-movies.net
imdb:
min_score: 7.0
min_votes: 5000
min_year: 1980
reject_genres:
- horror
- documentary
- musical
- biography
deluge:
host: localhost
port: daemonport
user: username
pass: password
movedone: ~/torrents/completed/Movies
ByteSized Setup: The Overview
As an opening article for this blog, we want to talk about our software and our whole setup. This will be first article in a whole series on the topic.
As for why we're writing this: We want to provide as much transparency as possible and help our (potential) members understand how the gears and cogs under the hood work. While this is the main goal, it would also be great to provide ideas for other people and their own setups. As we've encountered alot of interesting challenges along the way, it also might spare some people the trouble of reinventing the wheel. One of the best possible outcomes though would be to start a discussion on how we could improve our setup, so we're welcoming all constructive criticism and suggestions as well.
This first post should provide a a bird's eye view of the whole system, omitting the details to illustrate the general idea. Other articles will try and give an in-depth explanation of each of the separate parts.
The Servers
We have chosen to not give out VPS's but to use normal unix user accounts to share the servers. This way we feel like we can give better support and have some control over the servers, while still giving the user a certain amount of freedom. A box is simply a user account on a linux server with preconfigured software.
The Website
All things begin and end on the website. The website, written from scratch, running on Ruby on Rails uses a mysql backend for the user database. To communicate with the servers (e.g. deploy users to a server or get the harddisk usage) it connects to a daemon running on each server. For example: After a user is registered we, as admins, can choose a harddisk and server to deploy the user on through the admin-interface, when this happens the website sends a request from the website to the chosen server to the "byte-daemon".
The Daemon
The byte-daemon has several purposes one of them is creating new users, but disk usage and service status are a few other things it handles for the website as well. It acts as a man in the middle if you will. The byte-daemon is running as a dedicated account on the boxes, which has a few sudoers entries to allow it to do things that are usually reserved for root (adding/deleting users, etc.)
The Tools
The byte-daemon runs an other program called the "slicer". This slicer tool actually does the hard work, creates a new user account, creates a random secure password, selects ports for all the services, sets up quota and configures the torrent clients for the new user. When it is done it sends a request back to the website with the data it created. The website saves the data in the database and notifies the new user that his account is ready.
A picture says more than 494 words
Even the people who skipped most of the text should now be able to understand the first parts of our setup. Next week we will try and go a bit deeper into our software-stack.
Hello world!
The typical opening post of any good blog: "Hello World!". With this blog we want to keep our members, future members and everybody else in the loop about ByteSized, torrents, technology and development.
We will have posts adressing all these topics, some will give you some background information about ByteSized, some will be how-to's on a specific subject, and others might be just random rants.
We will try to get at least one blog post out each week.
We hope you will enjoy our writing.