Unplanned outage

Posted by on January 27, 2012 No comments

Was hoping to post about more exciting things that have happened but for the moment, our servers are down due to unplanned outage. We will post an update when we have the servers backup and running along with a technical summary.

Update: Back online and as promised an explanation.
The problem has a number of contributors mostly consisting of stupid mistakes that should not have happened but lessons well learned.

First a little background:
Main login server authenticates off an ldap server so we can do central authentication with many different services (lots of logins, one password/user data etc). SSH servers run nslcd, a daemon that queries nss details  from an ldap server (such as username, uid, shell, home directory etc). NSCD (different to nslcd), is a caching daemon that caches results from the ldap server.

Factors in downtime:
1:NSCD was turned off (caching daemon). By itself wasn’t that big an issue but still not recommended. The correct response was to turn it off only while debugging or to force the cache to update when we need it to update.

2: Debugging was enabled on the ldap server to a separate non syslog file with excess debugging information. I had turned it on to debug an issue and just left it on by accident.

3: /var/log or even /var /was not in a separate directory. Not anticipating logs to grow that fast before logrotate took care of it was my fault. openldap was logging directly to a file which didn’t go through syslog, getting around that point. I imagine that even if we had separated /var, it would have still caused issues as slapd (openldap server), still stores its real files for non special databases in /var/lib/ldap.

4: Even with all of these issues, it was processes on our main login server that started the ball rolling. When we tested it ourselves, because we weren’t causing the server to lookup much, we didn’t see much but as soon as people started using it and we started getting more members, even doing a simple ls -al of the home directories, causes the server to do 100s of queries, each for specific uidNumbers due to the amount of users and then of course, each user doing similar commands etc etc, you can see how it went.

Investigation

We didn’t actually know what the problem was at the start. The possible candidates came to mind: someone was brute forcing the server, someone was scraping or rapidly querying the webserver in such a way that caused it to query the ldap server. (wordpress is inter linked with ldap), something was misconfigured in nslcd on one of the servers, someone was querying our ldap server from inside our network excessively hard.

When I saw the hard drive was full on the ldap server, I took down the server, fired up a new vm, setup a separate /var/log partition and rebuilt the server. This was so I could properly examine the logs in case anything more malicious was at hand while letting the ldap service run. Pretty easy process with slapcat however, my documentation on rebuilding was slightly ambiguous on recent changes and caused me to take longer than I thought it would.

Mitigating the problem + future proofing it. 

/var/log now has it’s own separate partition. NSCD is turned on and will continue to be tweaked to get the right cache time. Logging verbosity has been reduced to a more sensible level and only uses syslog. I’ve correctly fixed/updated the documentation on rebuilding. In the case that something happens again, i’m going to write a script to build the server in one go without prompting to save some time as well . We’re also looking into better monitoring solutions that will give us rapider response times so we can react faster.

This is a little embarrassing I admit, but I’d prefer to keep you all informed about what’s actually going on. If anyone is interested and wants to know more, hit us up.

Mark

Trip to campuscon WIT form

Posted by on January 10, 2012 No comments

Please fill in this form indicating whether you’d like to come down for campuscon WIT with us. We will be posting up details and locations for transport and accommodation so check back with periodically.

https://docs.google.com/spreadsheet/viewform?formkey=dEtvRjgxaHlMYjhWOFVKWFMzUGJTbHc6MQ

Here’s what we’ve got so far
Transport: Bus – buseireann public bus return ticket – approx 13 euro (cheaper if you book online)

http://www.buseireann.ie/pdf/1307702391-4.pdf

Accommodation: single/double/triple bed (all for 49 euro a night). If you can get 3 people to split it, that’s 16.3 euro per person per night. (location tinyurl.com/ramadahotellocation)

As you can see from campuscon location (tinyurl.com/campusconlocation) it’s quite close

Conference cost – 5 euro on the door.

Note: campuscon link – http://campuscon.hackingwit.com/

WIT Campuscon hacking challenge

Posted by on January 9, 2012 No comments

WIT’s hacking society are putting on a full day event on 21st January where there will be a team hacking event http://campuscon.hackingwit.com/.

With some great feedback on facebook, we’ll be going down on 20th with bus eireann (12.60 euro a ticket if you book online). WITHacking are charging 5 euro into the event at the door. We’re still looking around for the best accommodation. We may also be able to subsidise part of the trip. Hope you can all come!

We want to organise a time we can all meetup to practice/train for the event and decide on teams to try our hand at the hacking. Remember, everyone’s a beginner, you’re guaranteed to have less fun and learn less if you don’t give it a try! This will be different to our regular workshops where it’ll be a meetup session where it is a group effort to practice and train rather than us preparing specific materials and you just following along.

Thursday 19th 12pm for a full day – official first day to come out and train / practice. Relaxed group orientated environment.
Requirements:
-Laptop
-Linux of some sort installed or on a cd (backtrack/ubuntu cds we handed out will do grand)
-Usb stick to install a persistent version might be handy too
-Network cable if you have one
-Power extension thinamabobs if you have them as well.
-Bring along some isos or vms of debian/ubuntu + windows if you have them for people who feel more technically comfortable.

To start off with, we’ll be focusing on web application vulns (most likely sql injection and further escalation from there) as it’s stated in http://campuscon.hackingwit.com/events.php.
If you want to get a jump start, check out http://www.hackthissite.org/ and http://sourceforge.net/projects/lampsecurity/ to begin with

Computer Security workshop on tonight! Next weeks sessions postponed

Posted by on November 29, 2011 No comments

Due to college assignments, we are going to postpone the second part of our Computer Security talk + workshop that we were going to run next week. The workshop will still take place today as planned. Hope to see you all there

Computer security talk Monday 28th

Posted by on November 23, 2011 No comments

We will be hosting a two part Computer Security talk on Monday 28th November 6pm following up with a workshop on Tuesday 29th November 6pm  and a part two the following week. Please bring laptops on Tuesday if you have them.

Update! We have a room, see the bottom of the post.

Leading up to the talk, we will be posting decoding challenges on twitter. Crack the code and send us in the answer and there could be some loot awaiting you Monday.

I will be diving right into the topic and show you real examples of how systems can be attacked and defended and talking about the penetration testing, a carear path where you are hired to professionally and legally, test the security of a system. We’ll be covering where to get started, books and mailing lists to follow and websites to visit.

We’ll be covering various attack methodologies and giving you a glimse of what’s inside malware along with popular tools that are often used to test security along with solutions to help you avoid becoming the next victim.

If that doesn’t sound exciting enough, we have built some systems we’ll be demoing a hack against live at the event and will be aiming to teach you how to do so and how to make sure when you’re writing code, that you don’t leave these holes open. If you were at our last PHP workshop, you will have gotten a taste of what you can possibly do. Come along to our security talk to learn how attacks are carried out and how to defend against them.

Dates and Locations

What: Computer Security Talk
When: 6pm Monday 28th November
Where: Kevin St KA-G-28

What: Computer Security Workshop
When: 6pm Tuesday 28th November
Where: Kevin St KA-1-16

What: Computer Security Talk (Part 2)
When: 6pm Monday 5th December
Where: Kevin St KA-G-28

What: Computer Security Workshop (Part 2)
When: 6pm Monday 6th December
Where: Kevin St KA-1-16

Integrated login to our website

Posted by on November 18, 2011 No comments

We have integrated our ldap authentication with our website netsoc.dit.ie (this one). All current members should be able to login. Give it a try!

https://www.netsoc.dit.ie/wp-login

Note logins do require https so it is as secure as https. We do have a signed https cert however due to firefox not having some required intermediate CA certs, it may appear invalid on firefox. We are working on supplying the right CA crt chain in apache and will fix  this very soon.

 

PHP Workshop 2!

Posted by on November 7, 2011 2 comments

We’ll be putting on another php workshop at 6pm KA-1-16 on Tuesday 8th November 2011.  We have fixed the unfortunate technical details that cropped up last week.

Again no prior knowledge require, bring laptops if you have them and yourself.

We have changed the structure of the workshop and will now be splitting into two groups. One that have had experience with programming before, and know some basic html and those who are in first year or have very little programming experience and know little to know about html.

Look forward to seeing you all there!

Motivated Individuals – We Need YOU!

Posted by on November 6, 2011 No comments

Roll up, roll up, read all about it! Netsoc are in search of two new committee members to assist in the day-to-day running of the society. If you have an interest in either of the following roles please send off an e-mail to net…@…ie, with a short description of why you think you are most suited for the position :

  1. Public Relations Officer
  2. General Committee Member
Neither of the roles require any experience/knowledge in the area of networking. The chosen candidates will be informed after the workshop on Tuesday night(08/11/2011) and will attend the next society meeting(09/11/2011 at 1P.M).
I could harp on about how being a member of a committee works wonders in a job interview but at the end of the day it helps build your skill-set tremendously, not to mention is far more fun than reading a textbook for hours on end in the hopes of understanding something which could have been casually explained through a committee e-mail(E.G. Setting up a Virtual Machine on our server).

Scheduled Down time periods

Posted by on November 2, 2011 No comments

As we are in in the process of building our services and servers up, we’ll be scheduling some downtime on servers every Friday from 10pm till 3am. In these hours, any of our services may be temporarily offline and servers may be restarted. This comes into effect Friday 4th November 2011 where we will be rebooting the login server.

PHP workshop Tuesday

Posted by on October 23, 2011 No comments

We’ll be doing a php workshop on Tuesday on basics of php, mysql and html. This is used primarily to make dynamic websites. No Prior php or mysql knowledge required.

When: 6pm Tuesday 25th Oct
Where: Kevin St, Annex building floor 1.

New members interesting in signup up welcome!