none
Load balancing with SQl server 2005 cluster in active/active configuration

    Pertanyaan

  • This is in regards to the MS SQL Server 2005 Cluster testing.

    I have the set-up in Active/Passive configuration.

    We decided upon moving to the Active/Active configuration, did some research on the internet for the additional features which we get with this configuration and what could be the benefits of this over the Active/Passive configuration. We have a query regarding the Active/Active configuration. Please find below the details on the same:

     

    The Current Setup in Active/Passive configuration has the following details:

     

    1)       There are two nodes “node 1” and “node 2” setup on Virtual Server 2005.

    2)       A cluster has been set up using “node 1” and “node 2”.

    3)       A SQL server default instance is installed on “node 1”. While SQL server 2005 was installed, it created a resource SQL server IP.

     

    To connect to this SQL server instance, our application uses SQL server IP created in step 3. (Please note that this is the only IP through which we are able to connect to SQL cluster). The fail-over features are working fine in this configuration. If “node 1” goes down then the SQL server instance runs in “node 2”. This is taken care of by cluster and is transparent to our application.

     

    However, there is no provision for load balancing as only one node is active at a time.

     

    After some research, we came across Active/Active configuration which is supposed to support load balancing.

     

    We understand that in this configuration, Step 1and 2 are similar to the Active/Passive configuration. The only difference is in step 3 where an instance of SQL server is installed on each node, thus providing two active nodes at a time. The failover works just like in Active/Passive configuration.

     

    As per the above information, the Active/Active configuration seems to be similar to two SQL server instances running independently.There will be two seperate databases and on failure of one instance other instance wont be able to cater to the requests designed for first instance, Thus providing no extra benefits from cluster.

     

    We require the information on how to take benefits of the load balancing features in this configuration.

     

    12 Maret 2007 5:17

Jawaban

  • Hi geetu...

    MSCS (Microsoft Cluster Services) is not a load-balancing product, it is simply a high-availability solution, period...load balancing does not come in to play at all with clustering a Sql database...the term Active/Active seems to imply that this would be the case, which is why we typically try to refer to them as Multi-instance clusters now instead of the Active/Active label.  Active/Active in the MSCS world basically means that you have 2 independent Sql Server instances running on 2 cluster nodes - these instances are independent of each other in all respects, obviously unless you link them in some manner with custom business logic, replication, etc.  Think of them for all intents and purposes as 2 seperate instances running on seperate servers at all times (as that's what they really are)...the only difference being that in case of a physical node failure (or service failure on a node), the instance will be moved to and hosted on a second physical server.

    I'd be curious to know what research you came across that implied load-balancing as a feature with MSCS so we can try and get it corrected, or possibly clarify what position the author was taking.

    To support load-balancing, or scaling out in a Sql Server environment you have a couple of different options, depending on your edition, environment, version of Sql, etc.  Take a look at the following articles for a start:

    http://msdn2.microsoft.com/en-us/library/aa479364.aspx

    http://www.microsoft.com/technet/prodtechnol/sql/2005/scddrtng.mspx

    HTH

    12 Maret 2007 16:44

Semua Balasan

  • Hi geetu...

    MSCS (Microsoft Cluster Services) is not a load-balancing product, it is simply a high-availability solution, period...load balancing does not come in to play at all with clustering a Sql database...the term Active/Active seems to imply that this would be the case, which is why we typically try to refer to them as Multi-instance clusters now instead of the Active/Active label.  Active/Active in the MSCS world basically means that you have 2 independent Sql Server instances running on 2 cluster nodes - these instances are independent of each other in all respects, obviously unless you link them in some manner with custom business logic, replication, etc.  Think of them for all intents and purposes as 2 seperate instances running on seperate servers at all times (as that's what they really are)...the only difference being that in case of a physical node failure (or service failure on a node), the instance will be moved to and hosted on a second physical server.

    I'd be curious to know what research you came across that implied load-balancing as a feature with MSCS so we can try and get it corrected, or possibly clarify what position the author was taking.

    To support load-balancing, or scaling out in a Sql Server environment you have a couple of different options, depending on your edition, environment, version of Sql, etc.  Take a look at the following articles for a start:

    http://msdn2.microsoft.com/en-us/library/aa479364.aspx

    http://www.microsoft.com/technet/prodtechnol/sql/2005/scddrtng.mspx

    HTH

    12 Maret 2007 16:44
  • Hi Geetu,

    Basically, Active/Active Cluster is two Active/Passive Clusters.

    As mentioned by Chad, they are completely independent of each other.

    HTH

    Jag

    15 Maret 2007 18:20
  • I hope you got more information from Chad's reply here and as suggested you might need another solution, cluster will not provide the load balancing.

    These 2 links shoudl give you information in this regard:

    http://www.microsoft.com/technet/prodtechnol/sql/2000/deploy/hasog04.mspx

    http://www.microsoft.com/technet/community/chats/trans/sql/sql0513.mspx

    http://www.sql-server-performance.com/dk_massive_scalability.asp

    19 Maret 2007 15:21
  • It's pretty clear what geetu is trying to get at. Here is your answer geetu:


    1. Instead of having the 2nd node idle until something happens to node1, you can split the databases between the 2 nodes.

    2. While this is not the definition of classic Load Balancing, it does help balance the load between the two nodes while they are both up.

     

    We have several customers who are configured like this and it works well.

     

    The downside? The “size” of each of the 2 servers in terms of CPU, memory etc must be enough to handle all the databases that are normally served from the 2 nodes in case one node fails.

    While this is the same requirement in the active/passive configuration, there is a danger that over time you will keeps adding load to both nodes independently and at time of failure, the failover node will die as well because of a sudden overwhelming load.

     

    Sometimes when smarty MSFT employees respond, they should be more respectful to the person who is posting a question, instead of being arrogant.

    30 Mei 2007 1:11
  • Nisim,

    I agree with you...from practical experience, this works and what you do is to create two clusters.  You can then partition the databases across the two servers and thus do a logical load balance.

    Down side of this approach in a simple two node configuration is that each node is a failover for the other....not the best case scenario.  Add a third node and designate that as the failover server for the two clusters.  That way, there is better protection.

    The only thing i don't agree with you is to call MSFT employees "smarties"...name calling is not a good thing.  From a MSFT perspective, i can see why they won't officially call this an "ACTIVE/ACTIVE" configuration because their chief competitor - Oracle, offers an industry strength, TRUE Active/Active cluster where both nodes of a two node cluster participate in the work and do load sharing.

    Hopefully, SQL 2008 will get us there?  Just a hope.
    18 Oktober 2007 16:09
  • What you guys are talking about isn't really load balancing either.  It sounds like you've created federated servers.  As the MSFT guy said, clustering is not load balancing.  It is simply a component of a high availability solution.  What is usually promoted now a days is what Microsft calls the N + 1 configuration. (N active nodes with 1 spare node)  For example, if you had an eight node cluster, you could have seven nodes set to actively provide different services and one node in standby mode should any of the seven nodes fail.

    By the way, while Oracle's RAC (Real Application Clustering) may support what you call "true" active/active clustering it still only supports one instance per server.  It also costs a fortune as it is an add-on and you have to pay a licensing fee for each core.  (The fee for just the RAC component is roughly the same as the SQL Server 2005 license.)  Microsoft gives you clustering as part of the operating system and they price their licenses per cpu not per core.  In other words, if you have a quad-core cpu you still only pay one licensing fee.  You probably don't want to go down the Oracle versus SQL Server trail because Oracle won't come out looking very good.  I am currently putting together a TCO (Total Cost of Ownership) study comparing SQL Server 2005 and Oracle 10g and Oracle is losing in a big way.

    This link is to an article discussing federating servers.

    http://www.sql-server-performance.com/articles/clustering/massive_scalability_p1.aspx


    18 Oktober 2007 20:33
  • Hi, Not sure if anyone is still interested in this post but, I just posted a comment on the sql-server-performance forum as well. I just got done releasing a BETA "Distributed" SQL Server Load Balancer at my website: http://www.poorstudent.org Let me know if this is what you guys were looking for. Thanks, Raj
    16 Februari 2008 17:42
  • To my understanding,

     

    When you configure Windows 2003 Cluster, it asks you which method you prefer - ie round robin etc.

     

    Round robin for example, distributes the requests between the machines in the cluster - one at a time. Will this not achieve load balancing amongst the nodes in the cluster by distributing the load. As well, if you have several SQL instances in a cluster .. the data would have to be replicated between the nodes. Otherwise how could a failover work - unless the data is syncrhonized on each node. Each node must be more than just an independent physical instance.

     

    If I am mistaken, please correct me.

     

     

    18 Februari 2008 17:28
  •  

    Hi Dwight, that’s a great question and you are absolutely correct; the data will need to be replicated between the (semi) independent servers.  J

    And that would be the only advantage you would get from Windows/SQL Server clustering. From what I know of Clustering Services (very little) you have two scenarios. In the first scenario; a 2-node cluster:  in which the 2 servers work either in tandem or to back each other up.  To get good performance you keep both the servers running, and one takes over when one of them takes a nose dive. In the other case you don’t get good performance because only one of your nodes is up at a time.  Another failure at this point will take your entire system down. On the other scenario you can have 4 servers at a time, where performance-wise you are getting 3 servers and 1 failure also. A second failure renders the Cluster useless.

    Not to mention in both these cases if the shared disks/SAN goes down, it takes down every node. That would mean, even if your public connection (HUB/Switch/BIG IP) to the system will be up, all your internal system will be down, for all your users to see. Also in both of these cases, you would have to use a single array of shared drives or a SAN. So your ultimate weakest link that slows down every SQL Server in the world, of disk contention, remains in place. I haven’t found a solution to disk contention issues yet. Not even with RAID 1 0 J, I have seen systems hit disk contention in large systems with sophisticated SANs.

    Also the one time I looked into a Cluster Service infrastructure, the IT director got a heart-attack from the bill we had put in front of him. The bills which easily run into a million dollars for the disk/SAN alone, not to mention extremely beefy servers, which also runs in Millions now-a-days, how is a medium to small size business supposed to pay for this? So to have a black-box system that gives you 1 failure fault-tolerance, you will have to shell out millions on Disk/ Server/SAN hardware and, enterprise licenses for SQL Server and Windows server (Data Center edition for 4-Node cluster), extremely skilled DBA’s and SysAdmins to maintain the architecture constantly.

    Given all that, “Distributed SQL Server Load Balancing” would allow you to run multiple small servers (100s if you need to). I am also looking at it as, "less waste of small outdated servers". All you need is a Standard License per server of SQL Server and Windows with CLR enabled in only of them at any given time. Now with all the little servers, you can have as many system failures as you like, as long as the middle-tier is up. All you have to do is de-activate the server that’s having problems. And if the middle-tier goes down, it’s extremely easy to re-create, (code is available at www.poostudent.org). Your performance issues are also taken care of because all these servers will be dividing the queries among them. Given that I haven’t come up with a good way to do true load-balancing but it’s in the works. For now it just goes round-robin like clustering would, so to speak.

    Also one of the biggest problems I see with Clustering is, troubleshooting. Query logging has been an issue with all database applications from the beginning. I used to work in OPS for a very large company at one time, and the developer would ask for the exact failures and we would have nothing for them. Because the Developers didn’t build logging into their system, and we couldn’t get good logging from mirroring or clustering for the specifics for an error. "Distributed Load Balancing" would allow you to log the entire query with the flip of a switch and automatically log errors.

    I see this as a white-box load balancing for small to medium, high availability databases, for small to medium size companies.

    Also I realize with the disks being shared in a Cluster service, the data doesn’t need to be replicated, which is a huge advantage for using Cluster Services like I said before. "Distributed Load Balancing" is really meant for reporting data at this point, which in most cases are created in bulk and sent to reporting servers, so all you should have to do is replicate that process multiple times, and would have current data in all servers. And since this is white box, and all the processes are available using T-SQL/OSQL the entire process of turning servers on/off during data deployment can be automated.

    Sorry it became a little longer than expected, and I realize this is a Cluster forum, I going to take this to the correct SQL Server forum. Thanks for question; it was great putting all this stuff on paper… Hope this makes sense… Thanks a lot.

    18 Februari 2008 20:50
  • I have been following this thread and have a few questions of my own. It will be great if someone can answer them -

     

    We have configured a two node SQL Server 2005 cluster. Along with SQL Server, we are also running Analysis Server on the cluster. We are running a single instance of SQL Server, so both the the SQL Server and AS services are running on the same node. Files for SQL Server databases and Analysis Server databases are on different SAN drives.

    We would now like the SQL Server service and Analysis server service to run on seperate nodes. Clustering for both services should stay.

    The node names are SQL1A and SQL1B.

    The virtual name of the SQL server instance is SQL1

    This way we can distribute the load on both machines and still have failover clustering.

     

    My questions are -

    1. Is it possible to distribute the SQL server service and Analysis Server service of the same instance on seperate nodes? i.e the SQL server service of SQL1 would run on node SQL1A and the Analysis Server service of SQL1 would run on node SQL1B. If yes, how can I do that?

     

    2. If the answer to the above is NO and I do have to install another instance of SQL Server (say SQL2); seeing that I want both services on seperate nodes, can I keep the Analysis Server service on SQL1 down and SQL Server Service of SQL2 down? The advantage of this would be that in the event of a failure, each node would still have one instance of SQL Server and one instance of Analysis Server running.

     

    Regards,

    Maulik

     

    P.S. Microsoft folks are welcome to reply. I do not believe the earlier answer was rude.

     

    10 Maret 2008 0:05
  • It is often overlooked the fact that SQL Server comes with a true load balancing capability available in ALL editions (including Express), with the know how to route related requests to the same node to avoid transaction replication and built-in support for high availability on each node. It costs nothing extra in licensing (other than the SQL instances licensing) and it scales by simple means of adding a new hardware and modifying one routing table, completely transparent to the application.
    The only issue is that this load balancing is not designed for the TDS protocol (ie. T-SQL requests), but for the Service Broker protocol: Service Broker Routing Examples. If you can design your application in terms of SSB requests rather than T-SQL requests, you will be able to leverage this built in load balancing.
    15 Maret 2008 8:57
    Moderator
  • What edition of SQL Server 2005 are you running on the 2 nodes?  Can you use Standard edition on both or do you need to run Enterprise edition?
    18 Maret 2008 18:09
  • I am trying to do exactly this with two new servers.  I will be migrating DBs from several existing SQL 2000 servers to one instance on the two-node cluster and I want to use the default instance name so the apps do not need to be tweaked.  I will install the 2nd instance with a name for future apps.  Can I have do this...have a two node cluster, two instances, both active and failing over to the other server and one instance with the default name?
    02 April 2008 21:18
  • Can one of the active clusters use the default instance name? 
    02 April 2008 21:22
  •  

    I am planning for a sql 2K5 clustering on active/active mode and wants to share dbs on both nodes.  I am very new to clustering and I do not know how should I install instances for this, can I install default instances on both nodes. can anyone guide me?

    thanks

    03 April 2008 22:12
  • Hello Mr. RajDatta,

            Here its clear to understand  what is “Distributed SQL Server Load Balancing”.
    I will request you, can you put the ways
    or steps/documnets/scripts  how to configure this “Distributed SQL Server Load Balancing”  among 2  Sql server 2005....


    Thanks You
     
    30 April 2008 5:42
  •  

    Hi Rabi,

    Sorry it took this long... Listen all the illustrations/instructions are posted at www.poorstudent.org. You will find white-papers and other stuff there as well. You can contact me from the "About Me" page, if you have any questions.

    Thanks,

    Raj.

    01 Mei 2008 14:40
  • I cut my teeth on Oracle long before Microsoft did much more than desktop. The difference in terminology when comparing Oracle and SQL Server can getcha. We are running RAC with multiple instances on each of two nodes in the rac servers. To Oracle a database is what's on disk and an instance is what's running, ie: the processes and memory on the server. Oracle instance DBPRD1 runs on RAC01 and instance DBPRD2 runs on RAC02. The database for both these instances is called DBPRD without a number. I also have instance DB1PRD1 on RAC01 and DB1PRD2 on RAC02. These access DB1PRD.

    You're right SQL Server is giving Oracle a run for it's money in terms of TCo but take a look at www.tpc.org and you will see that Oracle is still in the lead for performance. Of the top 10 databases, Oracle has six entries while Sql Server has 1.

    Having said all that, give serious thought to scaling up rather than out unless you have a high level of confidence to deal with the complexities of a true active/active load balanced configurations. I supported a seven instance oracle parallel server with 46 SAP servers accessing the seven instances using RVSD disk on a 5 frame SP2. This was for an oil & gas company with pretty deep pockets.
    28 Mei 2008 22:11
  • I cut my teeth on Oracle long before Microsoft did much more than desktop. The difference in terminology when comparing Oracle and SQL Server can getcha. We are running RAC with multiple instances on each of two nodes in the rac servers. To Oracle a database is what's on disk and an instance is what's running, ie: the processes and memory on the server. Oracle instance DBPRD1 runs on RAC01 and instance DBPRD2 runs on RAC02. The database for both these instances is called DBPRD without a number. I also have instance DB1PRD1 on RAC01 and DB1PRD2 on RAC02. These access DB1PRD.

    You're right SQL Server is giving Oracle a run for it's money in terms of TCo but take a look at www.tpc.org and you will see that Oracle is still in the lead for performance. Of the top 10 databases, Oracle has six entries while Sql Server has 1.

    Having said all that, give serious thought to scaling up rather than out unless you have a high level of confidence to deal with the complexities of a true active/active load balanced configurations. I supported a seven instance oracle parallel server with 46 SAP servers accessing the seven instances using RVSD disk on a 5 frame SP2. This was for an oil & gas company with pretty deep pockets.
    28 Mei 2008 22:12
  •  

    What about the scenario where the same databases are installed on the 2 nodes sharing the data files. No need for replicating data between the 2 nodes here, and true load balance as you have 2 separate cpu’s working at the same databases at the same time, just a matter of sending the request 50/50 to each node, someone mentioned the “Service Broker” it might come in handy here.

    01 Juli 2008 12:16
  •  

    When configuring a multiple Instance failover cluster - yes one instance can leverage the default instance name.  The remainder of the instances require a named.  For simplicity and consistancy, I name then all.

     

    Ross

    11 Juli 2008 12:31
  • Hi,

    Our organization is planning to go for Active/Active clustering resolution with SQL 2005. I have gone through various discussion boards on-line to find best practices for Active/Active clustering. It doesn't give me confidence to purpose Active/Active resolution to my director.

     

    Can anyone please share a concrete knowledge base (on any site or forum) that can give me Active/Active vs Active/Passive comparison / draw backs or advantages we can gain with Active/Active solution.

     

     

    Thanks

    08 Oktober 2008 17:02
  • Hi!

     

    Can someone please comment on the scenario below:

     

    We have a Cluster of three SQL servers running in Active/Active confirguration. In this scenario two servers are active running a single SQL instance and one is dedicated failover. Now to load balance this cluster we put a third party load balancer like barracuda or Zeus and create a load balanced Service for SQL. The load balancer will assign a virtual IP to the SQL Servers in the cluster and will load balance all queries on the two active nodes.

     

    Now first is this possible with SQL 2005 because i have done it with MySQL and it works fine.

     

    Secondly if this is possible what effect will it have on database consistancey of the cluster.

     

    Thirdly will it have desired impact on the performance.

     

    And also what kind of downfalls exist in this configuration especially when one of the Active/Active servers fail.

     

     

     

    20 Oktober 2008 16:15
  •  

    Yes, only the one instance within the cluster can use the default name.  Every other instance will require a dedicated instance name and different set up cluster resources.

     

    I hope this helps.

     

    Ross

    20 Oktober 2008 16:19
  • So load balancing is not officially supported with SQL Server clusters and Microsoft doesn't really support an Active / Active environment either based on what you are thinking about when using MySQL.  So basically, you are looking at a single instance or multiple instance failover configurations.  If you using two nodes in a multiple instance configuration, two nodes can be active, however, they are hosting completely independent SQL Server instances and databases.  For example, Node 1 can host SQL Server instance1 and node 2 can host SQL Server instance 02.  It is not recommended to load balance these machines.  In addition, if you do, and you are writing to the databases, you need to synchronize the  content.  Some people "bolt on" peer-to-peer replication to sync the independent DBs, however, that adds overhead and additional complexities. 

    Start with a single instance, add instances if necessary, however, remember databases are not load balanced between all nodes running in the cluster. 

     

    I hope this helps.

     

    Ross

    20 Oktober 2008 16:28
  •  

    KAISER,

     

    I have a dedicated chapter on Clustering including advantages and disadvantages in my SQL Server 2005 Management and Administration book.  Also in my upcoming SQL Server 2008 Management and Administration book. 

     

    SQL Server does not support Active / Active solutions.  Perhaps you are referring to installing more than one instance within your 2 node cluster and configuring a specific node to be an owner of an instance?

     

    That is supported.  Let me know if this is what you are looking for?

     

    Thanks

     

    Ross

    20 Oktober 2008 16:35
  • Chad Boyd had another response earlier in the thread explainging the differences -

     

    Hi geetu...

    MSCS (Microsoft Cluster Services) is not a load-balancing product, it is simply a high-availability solution, period...load balancing does not come in to play at all with clustering a Sql database...the term Active/Active seems to imply that this would be the case, which is why we typically try to refer to them as Multi-instance clusters now instead of the Active/Active label.  Active/Active in the MSCS world basically means that you have 2 independent Sql Server instances running on 2 cluster nodes - these instances are independent of each other in all respects, obviously unless you link them in some manner with custom business logic, replication, etc.  Think of them for all intents and purposes as 2 seperate instances running on seperate servers at all times (as that's what they really are)...the only difference being that in case of a physical node failure (or service failure on a node), the instance will be moved to and hosted on a second physical server.

    I'd be curious to know what research you came across that implied load-balancing as a feature with MSCS so we can try and get it corrected, or possibly clarify what position the author was taking.

    To support load-balancing, or scaling out in a Sql Server environment you have a couple of different options, depending on your edition, environment, version of Sql, etc.  Take a look at the following articles for a start:

    http://msdn2.microsoft.com/en-us/library/aa479364.aspx

    http://www.microsoft.com/technet/prodtechnol/sql/2005/scddrtng.mspx

    HTH

     

    20 Oktober 2008 16:38
  •  

    Ross,

    Thanks for update. It does clarify the active/active (multi instance) clustering resolution. We have Itanium server with 8 processors configuration and it has SQL 2005 running on it . Out director is planning to buy another one of same kind and have it clustered (data will be on SAN) . The requirements are to have it redundant and utilization for both servers.

     

    For Example

    We have 10 databases, we need to have 5 databases running form one node and 5 databases from other. But in case of failure for node 1, the 5 databases on node 1 should start sharing node 2 resources. We don’t want active/passive kind of configuration so one server is always 100% and other one is sitting there and doing nothing unless the failure occur for first one.

     

     

    Thanks

     

    20 Oktober 2008 18:03
  •  

    Ok that makes sense.  Just ensure that you tune the SQL server memory accordingly and that the surviving node - in the event of failure - can support both instances and 10 DBs on the same server.  Some individuals add an additional node, therefore, you have A/A/P. 

     

    Ross

    • Disarankan sebagai Jawaban oleh guptaajay1985 01 Desember 2009 10:14
    20 Oktober 2008 18:06
  •  

    Kalidasan and Nimish Load balancing feature has not the SQLServer.Now we have a critical time when 1500 users insert the data simultaneously at evening and morning.So what should I do I have empty servers but SQL Server not supporting Load balancing.

    And they all hit to one singledatabase???

    What the other companies does..in this case???


    Thanks, /\J/\Y GUPT/\ SQL DBA ..India..
    01 Desember 2009 10:15


  • Greetings.

    I'm sorry, but there is as difference between “arrogance”
    and “certainty”. There is also a difference between “can be done” and “It’s
    OFFICIALLY supported”. And finally and with all due respect, there is a difference
    between a “System Administrator” and a “Wise Smart A$$”.

    You see, just because you mentioned “We have several
    customers who are configured like this and it works well.”, that doesn’t mean
    it’s a correct configuration, and since the “Network Name” and the “IP”
    resources can only be at one Node at the time, how is it “Load Balancing”. It
    may “balancing the load on the DataBases from the CPU and RAM perspective, but
    it is still hitting the same storage, and the Server that will receive the
    requests from the application it’s the “Active Node” from the cluster, so…

    Although I’m not a “smarty MSFT employee”, I’ll reply with
    the same contend “MSCS (Microsoft Cluster Services) is not a load-balancing
    product, it is simply a high-availability solution, period.”

    You see, as considered by many to be a very experienced
    System Administrator, I have to deal with people like you all the time, the
    ones who just want to be smarter than the creators and supporters of the
    products themselves, just because you found a way be “configured like this and
    it working well.”, or so you say.

    Despite of everything I do hope my simple explanation
    will help to better understand the real purpose of MSCS.

    Best regards.

    16 Februari 2012 22:58