Spacewalk? Local Repo? Cache?

Home » CentOS » Spacewalk? Local Repo? Cache?
CentOS 17 Comments

I have a mix of CentOS 5, 6, and now 7 servers at work. There are enough of them now that it is starting to make sense for them to get updates from an internal source.

I’ve seen RHN Satellite in years past. It looks like it may be a way to allow Windows admins here (familiar with WSUS) to update Linux boxes. A local repo might be easier to set up, but (as with Spacewalk) it seems like we’d end up with a lot of packages we don’t need. A proxy and a sufficiently-large cache might do the trick if the first Linux box to get updates populates the cache which the files the others will need, but I haven’t looked into this enough to see if there’s even a way that works.

How do you all keep a dozen or more Linux boxes updated?

Thanks!

17 thoughts on - Spacewalk? Local Repo? Cache?

  • I was about to answer the question then all of a sudden my eye caught this footer which offended me, so I decided mention this fact instead of answering question…

    Valeri

    PS I never feel obliged to anything that is sent to me in e-mail without me originally soliciting it. All obligations lie purely on the sender. This has always be that way and will always be no matter whether I read your crap or not (sorry, everybody who does not send e-mail with that crap, it is really difficult to hold one’s feelings when you just got offended).

    ++++++++++++++++++++++++++++++++++++++++
    Valeri Galtsev Sr System Administrator Department of Astronomy and Astrophysics Kavli Institute for Cosmological Physics University of Chicago Phone: 773-702-4247
    ++++++++++++++++++++++++++++++++++++++++

  • I don’t think there is a way to do it that doesn’t take more human effort than it is worth unless you have limited internet access. It is basically designed not to work. A simple squid proxy with the file size bumped up will work with no extra attention (and be useful for all your internet accesses), but the first dozen or so runs are probably going to pick different mirror URLs instead of reusing the copy you have already cached. You can change the repo mirrorlist entry to a fixed system – but then your updates will break if it is down. Or you can mirror a bunch of stuff you’ll never need into your own repo. Or set up some special-case thing that only works for CentOS –
    or maybe even just one version of CentOS.

  • Chris Beattie wrote:
    We have over 170 servers and workstations. We use yum update for system stuff. If you really want an internal source, build a repo of your own.

    I installed Spacewalk in ’09. While I was doing it, it went from .3 to .4
    or .5 – don’t remember. For a dozen or so servers, it’s *vastly* more effort to install and configure, and presumably maintain, than you would spend if you just set up an internal repo.

    I agree with Varleri – this is a ludicrously long and extremely pissy postscript. I probably should have joined him in *not* responding, since a) it says “may you posted to a public email list, which is international in scope, and therefore a null and void statement, since it would *only* be applicable to someone who had signed an NDI, and b) you’re not offering anyone here to pay for such.

    mark

  • Hey Chris,

    If you are up for the challenge you can try a hybrid of squid + local repo. Local repo is based upon the basic nature of rsync which copies everything. You can write a script that will filter a list of urls of mirrors and will prepare a “fetch” list of files which will be fetched only the
    *rpm* from one of of couple mirrors into local repo. For each file it has in the cache it will first verify if the file exists in the local repo and if it is then it can redirect the client
    (transparently or with 302 redirection) into the local server.

    You can use do something similar with nginx to store the file permanently like in the idea of:
    https://code.google.com/p/youtube-cache/source/browse/#svn%2Ftrunk%2Fnginx

    The main issue would be the rpms while the packages sql\xml and other repo related stuff should be handled only by squid caching.

    Email me if it’s was interesting to hear about the idea.

    Eliezer

  • I guess my feeling will not hurt if I add my reply *here* ;-)

    We keep local mirror, which I’m pointing my CentOS boxes to. When I know some update is critical I kick the script that walks through all boxes and installs all updates accumulated by that time (yum clean all; yum -y update). In the past when I had awfully important servers under CentOS
    (they are FreeBSD now), I was testing updates on a separate box first to see if they will or will not break anything, and find the way to not have production stuff broken before actually install updates. I kick my script into action to the contrary to having daily, hourly or weekly cron job as I have system integrity verification system which will give me a kick every time anything changes without a reason. This makes cron job prohibitive for me (and requires me to incorporate that integrity stuff into update script, – which is beyond the scope here).

    Valeri

    ++++++++++++++++++++++++++++++++++++++++
    Valeri Galtsev Sr System Administrator Department of Astronomy and Astrophysics Kavli Institute for Cosmological Physics University of Chicago Phone: 773-702-4247
    ++++++++++++++++++++++++++++++++++++++++

  • That signature is time-wasting meaningless waffle (or gibberish, if one prefers). It has no legal worth despite the ‘efforts’ of the untrained legal expert who composed it :-)

    Regards,

    Paul. England, EU.

  • 2014-09-29 20:59 GMT+03:00 Chris Beattie :

    install yum-utils and use reposync to create local mirror with only newest packages.

    Lorem ipsum dolor sit amet, consectetuer adipiscing elit, sed diam nonummy nibh euismod tincidunt ut laoreet dolore magna aliquam erat volutpat. Ut wisi enim ad minim veniam, quis nostrud exerci tation ullamcorper suscipit lobortis nisl ut aliquip ex ea commodo consequat. Duis autem vel eum iriure dolor in hendrerit in vulputate velit esse molestie consequat, vel illum dolore eu feugiat nulla facilisis at vero eros et accumsan et iusto odio dignissim qui blandit praesent luptatum zzril delenit augue duis dolore te feugait nulla facilisi. Nam liber tempor cum soluta nobis eleifend option congue nihil imperdiet doming id quod mazim placerat facer possim assum. Typi non habent claritatem insitam; est usus legentis in iis qui facit eorum claritatem. Investigationes demonstraverunt lectores legere me lius quod ii legunt saepius. Claritas est etiam processus dynamicus, qui sequitur mutationem consuetudium lectorum. Mirum est notare quam littera gothica, quam nunc putamus parum claram, anteposuerit litterarum formas humanitatis per seacula quarta decima et quinta decima. Eodem modo typi, qui nunc nobis videntur parum clari, fiant sollemnes in futurum.

    eh.

  • Hi Chris,

    Either a local mirror or spacewalk will do what you want. I find at my site with relatively little but expensive bandwidth, the cost of disks is much less compared to download time. Hence, I initially just mirrored over rsync and now rsync the changes every day or more frequently as required. At that stage my local machines pointed to the local mirror over my LAN.

    FWIW my current disk usage is about 0.7TB and I’m mirroring:

    -) CentOS
    -) cygwin
    -) dell
    -) epel
    -) rpmforge
    -) spacewalk

    After that, I then moved to spacewalk to manage the 30 or so CentOS
    machines currently in production. The effort to set up and maintain was not that great and the GUI front end is great for snapshots of the current state of my machines. Nice reporting tool for management.

    Currently I’m also moving into the OpenSCAP interface of SpaceWalk to provide the compliance reports that my company is starting to require. We do non-military civil engineering type work for government and its surprising the trickle down security and audit requirements being pushed down. I know that this can all be scripted but with a little set up its surprisingly easy via the GUI.

    Another big plus for me is that I love the local mirror that also makes spacewalk simpler. We do a bit of R&D so find when testing new servers a kickstart off the local http mirror is really quick. Initial application deployment on the kickstarts come directly off http – as previously mentioned if you run a local squid instance here this can be even faster. Next, the first step in my %POST of the kickstart is a couple of lines to disable the native repos and connect to SpaceWalk. From there all packages are deployed off SpaceWalk but still its over http so squid may still speed things up.

    The big move to make SpaceWalk viable for me though, was a few years ago when it fully supported PostgreSQL over Oracle. I didn’t have an Oracle license and the free version maxed out with three CentOS channels covering both x86_64 and i386 architectures.

    Finally, as a number of my developers are and want to continue to use Ubuntu/Debian, now that SpaceWalk supports debian packages, I’m looking at starting to mirror those channels and publish via SpaceWalk as well for auditing purposes. My devs have a lot of freedom on their own platforms, so if I can at least have an overview of their status that helps me.

    I also mirror EPEL. And publish it via SpaceWalk for all the same reasons.

    Hope that helps,
    -pete

  • How big is EPEL? And when you mirror with SpaceWalk does it preserve old version so you’d have the possibility to downgrade after a change?

  • If you maintain official public mirror you can not step away from original, you should always rsync –delete [other options]… and have exact replica of original. Do I miss something?

    Valeri

    ++++++++++++++++++++++++++++++++++++++++
    Valeri Galtsev Sr System Administrator Department of Astronomy and Astrophysics Kavli Institute for Cosmological Physics University of Chicago Phone: 773-702-4247
    ++++++++++++++++++++++++++++++++++++++++

  • My current EPEL mirror is 115GB. I mirror that from a local Australian mirror (AARNET) so I’m a little behind in time but my ISP gives me free volume to AARNET.

    Yes. To confirm in my EPEL 6 x86_64 channel I have, for example, clamav versions from clamav-0.97.8-1.el6.x86_64 to clamav-0.98.4-1.el6.x86_64.

    Interestingly SpaceWalk is quite an efficient store. For example:

    /var/satellite: 135GB

    where spacewalk stores all its packages for every channel. Verses the raw, rsynced mirrors:

    du -sch CentOS epel rpmforge spacewalk
    116G CentOS
    115G epel
    53G rpmforge
    5.6G spacewalk
    289G total

    I suspect that there are a lot of duplicates in the mirrors that spacewalk is smart enough to link to rather than copy. For example noarch rpms that appear in multiple channels. Also, those mirrors include the DVD isos, which are not in spacewalk.

    To save space on the raw mirror I have rsync set to delete after the package is removed from upstream but I don’t automatically delete from spacewalk. There are third party perl scripts to do this but I have lots of disk space on that server so have not bothered yet

    Cheers
    -pete

  • Sorry, I misinterpreted what you said. I thought you meant that you kept a local mirror of EPEL for use with your SpaceWalk-managed systems.

  • I haven’t used this but I have been following closely of late…..

    If you are after more than a local copy, have a look at pulp http://www.pulpproject.org/ it is part of RedHat’s next incarnation of Satellite, Satellite v6. talkes a lot of the hard work out of keeping a local cache of packages. If you want advanced features, you can install an agent on each machine to allow you to manage installs from the server.

    Redhat are moving to a group of products to replace satellit v5, consisting of Puppet, Foreman, Katello, Pulp and candlepin. http://rhsummit.files.wordpress.com/2014/04/caplan_t_0330_red_hat_satellite_6_overview_roadmap__demo.pdf

    Grant

  • I just wanted to say thank you to everyone at once for the advice. It’s much appreciated.

    I also wanted to apologize in advance to those who were injured by the legal text at the end of my previous message (I have no control over it. It is mandatory on all outgoing messages AND the use of personal e-mail accounts is prohibited where I work.), because, well, you’re just going to get offended again if you keep reading past the signature block separator. We all know the legalese down there is worse than YouTube comments, which at least have a nonzero chance of being funny going for them.

    Or, you need to try harder if you want to be as offended as the Cygwin mailing list is at these sorts of things.

  • Chris Beattie wrote:
    I hope you got a chuckle out of my final disclaimer – the quote from my
    …late… wife.

    Let me also note that, as I said, I am not allowed to speak for my agency or my company… which is why I’m on this list using my own “business”
    email (as opposed to the personal one, both from the same hosting provider).

    mark “no auto-sig here”

  • I’m usually the one who reacts sharply to that [il]legal nonsense in e-mail footers. I usually write my position about it to the best of my English language abilities, so the sender of the message has somebody’s else writing about legal nonseseness of it which he/she can pass over to the ones in charge who ordered the footer and hence can revert the decision. And I do my best to indeed reveal the feeling of the person who does read that footer and comprehends what is being said, thus to encourage the sender to indeed pass the message on up the ladder…

    Cheers ;-)

    Valeri

    ++++++++++++++++++++++++++++++++++++++++
    Valeri Galtsev Sr System Administrator Department of Astronomy and Astrophysics Kavli Institute for Cosmological Physics University of Chicago Phone: 773-702-4247
    ++++++++++++++++++++++++++++++++++++++++

  • On 9/30/2014 3:14 PM, m.roth@5-cent.us wrote:> I hope you got a chuckle out of my final disclaimer – the quote from my

    It’s a good one. I had to go back and take a second look because I stopped reading at the double dashes, heh heh.

    I don’t know how the footer disappeared (Honestly, I despise them as much as anyone else!), but I figured one more message was worth the risk before I re-lurk.