What's the best way to leverage a Technical Advisor when migrating from LAMP to a Big Data stack?

Abbas Alidina Founder & CEO at Crowdbabble - Simplifying Social Media Analytics.

November 16th, 2014

We've reached an agreement with an experienced CTO/Technical Advisor to provide 1-2 days of support on a monthly basis in order to help us scale our back end. For context, our startup Crowdbabble is a social media analytics service.

Crowdbabble currently uses a LAMP stack, which was acceptable when we had a handful of paying customers. Now that our paid customer base is growing, we need to ensure our back-end is scalable before we outgrow our current setup. We are using outsourced developers to get to where we are today.

From a technical perspective, are there any do's, dont's or best practices on what to prepare or provide to this Technical Advisor (such as data models, etc.)? We want him to hit the ground running and ensure we can make the best use of his time.

We're also concerned that there may be a gap between the advise we receive from our Technical Advisor vs the capabilities of our developers (who only have LAMP experience).

Are there any tips on how to reduce this gap and how to make the best use of the Technical Adviser?

Sam McAfee Building Popup Incubators for Corporate Innovation Programs

November 16th, 2014

Yeah, @Abbas. I think the whole framing of the question is looking at it the wrong way. There several issues here.

1. Facebook uses PHP, so let's not pretend it doesn't scale. It's not the language, but the architecture of the system is that is going to make the biggest differences.

2. You're using outsourced developers for all your development? You have bigger problems than making the most of your advisor's time. You are a software company. You need to have a strong internal team, no matter what. You can't cobble together a scalable system with an advisor and an outsourced team.

3. Scaling is not a one-time event. You should never do the big-bang rewrite if you can possibly help it. The way you approach these issues is by attacking one bottleneck at a time. And you really need someone who's done it before to be on the team doing it with you.

I'd be happy to chat further about it if you're interested.

Bob Binder Member of Technical Staff at Software Engineering Institute | Carnegie Mellon University

November 17th, 2014

You’ll very likely be running your big data systems on a LAMP stack, so beware of any advisor that claims to be able to help you “migrate” to your new capability.

I wonder why you find this stack inadequate - this technology has and is used for many complex high-load transaction processing systems hosted in "the cloud".

As far as choosing an advisor, why bother? I'd look to hire a competent technician who has a proven track record building apps that work and pay them for results, not advice.

Lawrence Lerner Digitalization and Transformation Coach

November 16th, 2014

@Abbas, great discussion topic. If someone is providing you with technical advice, I would start by askingthem, what they would like to see. It will tell you a great deal about their ability to ask smart, consultative questions. You'll also learn their breadth of knowledge.

Some things they should ask for:

Your current technology stack
Datamodel
Any architectural diagrams or artifacts (technical design specifications, functional, user stories)
Access to a sandbox environment to play around with your existing product
Your standards for Quality Assurance and software testing
Coding and development - tools, standards
Access to your current technical lead

As @Vladi has mentioned, LAMP is highly scalable too. Apache, Linux, MySQL, PHP/Python or other Open Source software runs very large enterprise class implementations. I've built quite a few corporate systems that support millions of users and hundreds of thousands of transactions per hour. Ping me if you need LAMP examples.

Cheers

Abbas Alidina Founder & CEO at Crowdbabble - Simplifying Social Media Analytics.

November 17th, 2014

Wow. Some very helpful insights!

I wasn't sure how scalable LAMP is, but it looks like we don't necessarily need to migrate away from LAMP. Will explore ways to optimize our architecture.

@Vladi - thanks, we will look into ScaleBase and see how we could leverage this. Will ping you if I have any questions.

@Lawrence and @Sam - I have taken you both up on your offers and just sent you a message to move forward.

Thanks everyone,

Abbas

Will Koffel Co-Founder at Outlearn

November 16th, 2014

I'll echo something Lawrence said: a good technical advisor should be pro-active in bringing their expertise to bear. They will have a solid set of questions that give them a comprehensive picture of the strengths and weaknesses of:

- your technology stack

- code and architecture

- development process

- infrastructure

- team

Once you've got an agreement with them, you should ask them to share that with you in advance, and ask them what's helpful.

I have a standard "health-check" of sorts that I do all the time for companies in this situation. It gets into the technical detail of library choices, deployment methodologies, vendor selection, and project management tools, but really just because those details tend to illuminate whatever the core issues are that are preventing a team from reaching their full-potential.

Amandeep Khurana Principal Solutions Architect at Cloudera

November 17th, 2014

You have some great answers on this thread already. I work with a lot of customers on advising them on scaling infrastructure, data platforms and often times migrating from traditional stacks to some of the newer technologies. There are three aspects to the advice I give customers:

1. In most cases, the new stacks start out as complementary and over time replace the older stack for certain use cases. It's not a complete rip and replace. There are workloads and use cases that are better suited for the LAMP stack and they should stay there.

2. In almost all cases, the migration is limited by the teams' understanding of the new technologies and their comfort level in supporting it with high level of reliability. This turns out to be an education effort as much as a technology migration effort.

3. The overall migration strategy needs to take into account the above two, goals of the overall product, deadlines etc and your advisor should be able to navigate you through these considerations and maybe even help with part of the effort depending on the agreement you have with them.

Hope this helps.

Amandeep